Troubleshooting and Upgrading AD FS Farms

, ,

Upgrading an AD FS farm is usually a straightforward and easy task. There are many blogs detailing the process from Server 2012R2 to Server 2016/2019. Here are the general steps for upgrading a farm.

  1. Setup up a new server and install the AD FS role.
  2. Add the server to the existing farm.
  3. Set the new server as the primary server.
  4. Point the other servers to the new primary server.
  5. Install additional servers to the AD FS farm.
  6. Uninstall AD FS on the old servers to remove them from the farm.

What are your options when this process doesn’t work? In my case, I could not add a new server to the farm. PowerShell and the GUI both returned errors when attempting to add a new Windows Server 2019 to the farm. I started looking at troubleshooting options and eventually decided to proceed with an attempt to in-place upgrade the Server 2012R2 farm to Server 2019.

Troubleshooting

Microsoft provides some powerful tools for troubleshooting AD FS issues. AD FS Help provides several troubleshooting guides and diagnostic tools that can help resolve issues with your AD FS farm. The tools are located here: https://adfshelp.microsoft.com/.

The AD FS Diagnostic Analyzer tool can provide a health check for you AD FS farm. To use the AD FS Diagnostic Analyzer, you need to install the AD FS Toolbox PowerShell Module.

Install-Module -Name ADFSToolbox -Force

Import-Module ADFSToolbox -Force

Once the AD FS Toolbox is installed, run the Export-AdfsDiagnosticsFile command which will generate a JSON file for upload. The command will run against the local AD FS server unless the farm is Windows Server 2016 or higher. You can also list the servers with the -adfsServers parameter. The -adfsServers parameter is required for 2012R2 farms. Upload the JSON file to the https://adfshelp.microsoft.com/ site and the site will display the Health Test Results. The site will detail any problems and offer step by step guides or links to documentation to remediate the issues.

Backup and Restore

There is also the AD FS Rapid Restore Tool found here: https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/ad-fs-rapid-restore-tool. The tool backs up the following AD FS configuration data:

  • AD FS database
  • Configuration file
  • Automatically generated token signing and decrypting certificates and private keys
  • SSL certificate and any externally enrolled certificates and corresponding private keys
  • The custom authentication providers, attribute stores, and local claims provider trusts that are installed

Unfortunately, the Restore Tool only supports a restore to the same version as the backup. This means that you cannot use this method to restore AD FS to a newer version of AD FS. You could restore the AD FS farm to a new set of servers and attempt to add upgraded servers to the farm as outlined above.

Other Tools

The https://adfshelp.microsoft.com/ site provides a number of other tools and support options. There is a full list of AD FS event items for 2012R2/2016/2019 with ID, Name, and Description. Claims X-Ray assists with debugging claims issues in you applications. The AD FS Event Module provides tools to gather and review the events from multiple servers. There are several more tools available as well.

In-Place Upgrade

Even after running the diagnostic tools several times and making the recommended changes, I still was unable to add a new server to the existing farm. The other option is to attempt an in-place upgrade of the servers in the AD FS farm. Technically, this is not supported as upgrading Windows Server with AD FS installed will uninstall the AD FS role. Prior to attempting this method, I made a snapshot (Hyper-V virtual Machines) of the AD FS servers and a backup of the AD FS farm’s current state. I started by upgrading the secondary AD FS server first. If I was unable to add the server back into the farm and promote it to the primary server, then my plan was to use the AD FS Restore Tool and rebuild the farm. These are the steps I took for the in-place upgrade.

  1. Upgrade the secondary AD FS server to Server 2019.
  2. Install the AD FS role.
  3. Add the upgraded server back into the farm.
  4. Set the server as the primary AD FS server in the farm.
  5. Verified that AD FS was still working for our services.
  6. Upgrade the former primary server, reinstall the AD FS role and set it as the primary server.

I also did an in-place upgrade of the Web Application Proxy server and added it to the farm with the Install-WebApplicationProxy cmdlet. The final step is raising the farm functional level with the Invoke-AdfsFarmBehaviorLevelRaise cmdlet. This will enable the new features Verify the update completed with the Get-AdfsProperties | Select CurrentFarmBehavior cmdlet.

While it is not a supported option, the in-place upgrade of my AD FS farm worked perfectly. Hopefully, it is not an option you will need.

 

Ben Dumke

@BenDumke

SCCM Reboot DECODED:: How to make a PC Cancel, Start, Extend or Change mandatory reboot to non-mandatory on the fly.

, , , , , , , , ,

How to make a PC Cancel, Start, Extend or Change mandatory reboot to non-mandatory on the fly.

In the past to stop a PC from rebooting when you didn’t want it to people would stop the ccmexec service and do a shutdown /a

But here is the problem with that..

  1.  Shutdown /a will normally tell you that no shutdown is pending.
  2.  Any number of things can restart ccmexec.

First, how to check if a reboot is pending VS a reboot is GOING to happen

On a win10 PC or has powershell 5 installed, use this

#Detect pending reboot:
Invoke-CimMethod -Namespace root/ccm/ClientSDK -ClassName CCM_ClientUtilities -MethodName DetermineIfRebootPending

On a win7 or old powershell 2 you need to use these.

#
([wmiclass]'ROOTccmClientSDK:CCM_ClientUtilities').DetermineIfRebootPending().RebootPending
([wmiclass]'ROOTccmClientSDK:CCM_ClientUtilities').DetermineIfRebootPending().IsHardRebootPending
([wmiclass]'ROOTccmClientSDK:CCM_ClientUtilities').DetermineIfRebootPending().RebootDeadline
([wmiclass]'ROOTccmClientSDK:CCM_ClientUtilities').DetermineIfRebootPending().NotifyUI

It will spit out something like this..

#PC with no pending reboot.

DisableHideTime     : 12/31/1969 2:00:00 PM
InGracePeriod       : False
IsHardRebootPending : False
NotifyUI            : False
RebootDeadline      : 12/31/1969 2:00:00 PM
RebootPending       : False
ReturnValue         : 0
PSComputerName      :

#PC with pending NON-mandatory reboot.

DisableHideTime : 12/31/1969 12:00:00 PM
InGracePeriod : False
IsHardRebootPending : False
NotifyUI : True
RebootDeadline : 12/31/1969 12:00:00 PM
RebootPending : True
ReturnValue : 0
PSComputerName :

#PC with pending mandatory reboot, notice the time stamp.

DisableHideTime : 5/1/2019 3:28:56 PM
InGracePeriod : True
IsHardRebootPending : False
NotifyUI : True
RebootDeadline : 5/1/2019 11:28:56 PM
RebootPending : True
ReturnValue : 0
PSComputerName :

 

Now for the magic…..

Everything that SCCM uses for knowing when and if it should reboot comes from here.

HKLM:\SOFTWARE\Microsoft\SMS\Mobile Client\Reboot Management\RebootData

On a PC with no reboot pending, this key is empty.

So starting with that, this is how we can CANCEL a pending reboot.

#CANCEL a pending reboot
Remove-Item -path 'HKLM:\SOFTWARE\Microsoft\SMS\Mobile Client\Reboot Management\RebootData';
Remove-Item -path 'HKLM:\SOFTWARE\Microsoft\SMS\Mobile Client\Updates Management\Handler\UpdatesRebootStatus\*';
Remove-ItemProperty -name * -path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired';

#on PS2.0, "Remove-ItemProperty" doesn't work, so use this.
#Remove-Item -path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired';

shutdown -a 
Restart-Service ccmexec -force

But what if you just want to cancel the mandatory reboot and change it to a non-mandatory reboot so the user will still get the popup telling them they “need to” reboot?

#change mandatory reboot to  non-mandatory reboot
Set-Itemproperty -path 'HKLM:\SOFTWARE\Microsoft\SMS\Mobile Client\Reboot Management\RebootData' -name 'RebootBy' -value 0;
Restart-Service ccmexec -force

What if we just want to extend the time of a mandatory reboot?

Example: In your client settings you have your reboot countdown set to 10 hours…. A user calls and says it’s going to reboot in 10 min and needs it extended… This will reset that users countdown timer back to 10 hours.

#Reset SCCM reboot countdown timer.
$time = [DateTimeOffset]::Now.ToUnixTimeSeconds()
Set-Itemproperty -path 'HKLM:\SOFTWARE\Microsoft\SMS\Mobile Client\Reboot Management\RebootData' -name 'RebootBy' -value $time;
Restart-Service ccmexec -force

What if you want to kick off the built in SCCM reboot WITH the client settings countdown timer?
NOTE: Setting $Time to 0 will popup the non-mandatory reboot window.

$time = [DateTimeOffset]::Now.ToUnixTimeSeconds()
New-ItemProperty -LiteralPath 'HKLM:\SOFTWARE\Microsoft\SMS\Mobile Client\Reboot Management\RebootData' -Name 'RebootBy' -Value $time -PropertyType QWord -Force -ea SilentlyContinue;
New-ItemProperty -LiteralPath 'HKLM:\SOFTWARE\Microsoft\SMS\Mobile Client\Reboot Management\RebootData' -Name 'RebootValueInUTC' -Value 1 -PropertyType DWord -Force -ea SilentlyContinue;
New-ItemProperty -LiteralPath 'HKLM:\SOFTWARE\Microsoft\SMS\Mobile Client\Reboot Management\RebootData' -Name 'NotifyUI' -Value 1 -PropertyType DWord -Force -ea SilentlyContinue;
New-ItemProperty -LiteralPath 'HKLM:\SOFTWARE\Microsoft\SMS\Mobile Client\Reboot Management\RebootData' -Name 'HardReboot' -Value 0 -PropertyType DWord -Force -ea SilentlyContinue;
New-ItemProperty -LiteralPath 'HKLM:\SOFTWARE\Microsoft\SMS\Mobile Client\Reboot Management\RebootData' -Name 'OverrideRebootWindowTime' -Value 0 -PropertyType QWord -Force -ea SilentlyContinue;
New-ItemProperty -LiteralPath 'HKLM:\SOFTWARE\Microsoft\SMS\Mobile Client\Reboot Management\RebootData' -Name 'OverrideRebootWindow' -Value 0 -PropertyType DWord -Force -ea SilentlyContinue;
New-ItemProperty -LiteralPath 'HKLM:\SOFTWARE\Microsoft\SMS\Mobile Client\Reboot Management\RebootData' -Name 'PreferredRebootWindowTypes' -Value @("4") -PropertyType MultiString -Force -ea SilentlyContinue;
New-ItemProperty -LiteralPath 'HKLM:\SOFTWARE\Microsoft\SMS\Mobile Client\Reboot Management\RebootData' -Name 'GraceSeconds' -Value 0 -PropertyType DWord -Force -ea SilentlyContinue;

 

NOTE:: In all my tests after you run The powerShell command it takes about 30 seconds for the client to respond since after you restart the service it has to do all it’s internal checks to figure out what’s new.

#WeaponizedAutismFTW

DHCP Failover and Recovery

,

I recently had an issue with a Domain Controller that was unresponsive at a remote site. Since it was just a virtual machine with no special settings, I punted and rebuilt the server. After I did all the recommended steps for removing a DC from Active Directory and was already configuring the new server, I remembered that I had setup DHCP Failover for this site. I would be lying if I did not say a twinge of panic set in at that moment. I had setup DHCP Failover in hot standby mode almost five years ago and had not not given it a second thought. Fortunately, the process was so simple that I did not even need to use PowerShell. I deleted the old Failover Relationship and then recreated it. The detailed steps are below.

Delete Old Failover Relationship

First on the active DHCP server, open DHCP Manager. Right click on IPv4 and choose Properties

On the Failover tab, select the failover relationship and click Delete.

Click OK to confirm the deletion.

The Failover Relationship is deleted. Note that you will see failure messages if the partner server cannot be reached. Click Close.

Create New Failover Relationship

Now that the old failover relationship is removed, we can recreate it. Obviously, you will need to repair or rebuild the DCHP failover partner server first. Once the partner server is ready, open the DHCP Manager on the active server. Right click on the scope and choose Configure Failover.

Select the Scopes and click Next.

Click Add Server.

Type the server name or select it from the list of authorized DHCP servers. Click OK.

Click Next.

Configure Failover Options for your environment. You may need to choose a new Relationship Name if the previous DHCP Failover was not successfully deleted. Click Next.

Confirm the settings and click Finish.

Once the DHCP Failover is created click Close.

DHCP Failover is easy to configure and easy to recover. I was surprised at how smoothly I was able to add and delete the partnerships using Windows Server 2016 and 2019. I repeated the process several times as I wrote this article. I would suggest creating a DHCP failover partner especially if you have a remote site with little or no IT support staff. I was fortunate that I had setup DHCP Failover at this site several years ago. Of course, in my case better monitoring software would have helped alert me to the server failure sooner. Hopefully, I will be writing about installing and configuring monitoring software soon.

You may need to rely on PowerShell to remove a Failover Relationship. The commend Remove–DhcpServerv4Failover will delete the specified partnership. A complete list of PowerShell commands can be found here:

https://blogs.technet.microsoft.com/teamdhcp/2012/12/18/dhcp-failover-using-powershell/

 

Ben Dumke

@BenDumke