Disconnected WSUS – The “fun” of importing updates

, , , ,

Air gap networks have their own special challenges…

Hello,

My name is Charles, I went to the Midwest Management Summit back in May 2019. It was my second time attending the conference. You might remember me as “that guy who has no remote users” 🙂

I said I would blog about a tip I talked about shortly after the conference… This blog post is about 7 months late, I’m sorry!

So for those that do not know this conference, there is a session where anyone can go up on stage and present an IT-related tip or trick.

I went on the stage to talk very briefly about a rather important detail that you need to know when you deal with a disconnected/offline WSUS server: You need to track your update approvals on your internet-connected WSUS server.

On your disconnected WSUS server, you must not approve an update that was not approved on your internet-connected WSUS server or you will have issues.

Basics

Microsoft’s official documentation regarding setting up a disconnected WSUS instance can be summarized with the following 3 steps:

  1. Matching advanced configuration:
    • Express updates: If you want to use express updates, make sure that both the internet-connected WSUS and the disconnected WSUS instances have the same setting configured
    • Languages: Make sure that you select the same languages for update files
  2. WsusContent: This one is simple, just copy the “WsusContent” folder on the internet-connected WSUS to the disconnected one.
  3. Metadata: Export & Import the WSUS metadata with wsusutil.exe import/export

The problem

OK, so you made sure that you have the same advanced configuration, you copied the “WsusContent” folder and you imported the metadata from your internet-connected WSUS server… now what?

Well, now you have a bunch of unapproved updates in your WSUS Console on your disconnected WSUS Server. You don’t know which ones were approved or declined on the internet-connected WSUS.

The next part is from my personal experience… If you approve a bunch of updates and some are missing the associated content, WSUS gets stuck. In your WSUS console, it will show that you have thousands of updates “needing files”. What’s happening here is that you approved updates for which your disconnected WSUS does not have the content. In a normal scenario, WSUS would simply download the content from Microsoft. In our case, the WSUS server simply gets stuck there because some updates are missing files. And for some &*%!$% reason, WSUS will not skip over and verify the other updates once it gets stuck with a couple of updates missing content.

This is what happens when you approve updates that your disconnected WSUS does not have the content.

The solution

Make sure that the same approvals are mirrored on your disconnected WSUS instance.

And when I say update approvals, I mean which updates were declined and which updates were approved.

“OK, I will create the same automatic approval rules on my disconnected WSUS. Done.”

Sure, that might work if you don’t do any kind of cleanup on your WSUS instance. There are various solutions online of scripts that people use to decline/delete updates they don’t need (Itanium, Ia64, superseded updates, etc.)

The automatic approval rules criteria in WSUS are very basic and you will end up approving updates on your disconnected WSUS that are declined on your internet-connected. Ask me how I know…

I personally use Bryan Dam’s software update maintenance script, see his blog posts here and here. This script was originally written to maintain SCCM software update point WSUS instances but later he added the WSUS Standalone mode which I’m using for my WSUS Servers.

I won’t go into details here about which updates I’m declining and whatnot. Just know that if you use a WSUS Server, you should probably have some sort of maintenance script running regularly or you will have a bad time…

“How am I supposed to keep track of all the updates that I declined/approved?”

PowerShell.

With PowerShell we can get the information we want and it’s possible to script this so that we can export and import the update approvals.

When you’re ready to do an export of your internet-connected WSUS, you will have to export the metadata, copy the “WsusContent” folder and also get the list of which updates were approved. Make sure you copy all three at the same time to make sure that you have the matching metadata, content and update approvals.

My script

So where I work, the team responsible for copying content over to the disconnected WSUS server is different than the team maintaining WSUS. A procedure was written about how to perform the export and import process but it never worked really well and WSUS crapped itself… many times.

I decided that I had enough with this and tried to automate the process as much as I could.

I wrote a script that I called “Invoke-WSUSImportExportManager”

At first, I simply wanted to automate the following:

  • Exporting
    • Copy “WsusContent” to $folder
    • Copy the WSUS Metadata to $folder
    • Record WSUS information in a XML file and copy to $folder
      • WSUS Configuration
      • WSUS Computer groups hierarchy
      • WSUS Update approvals
  • Importing
    • Copy “WsusContent”
    • Import the WSUS Metadata
    • Update WSUS Configuration
      • Match the configuration
      • Re-create the same computer groups hierarchy
      • Approve the same updates to the same computer groups

And then it became bigger and bigger…

On top of doing the import and export process, it also does the following tasks:

  • Reindex the WSUS Database
  • Adds or Removes the custom indexes (Taken from Bryan Dam’s script)
  • Sets a couple of common IIS settings for the WsusPool that should be changed from its default values
  • Show locally published updates (third-party updates) in the WSUS console

All these “Actions” that the script performs are customizable.

The script is available on GitHub here. I tried to explain how the script works in the readme but feel free to ask me any questions if you need more information.

Note: My PowerShell skills are not super awesome so I’m sorry for the state of the scripts. If you have some feedback/suggestions regarding that, please let me know and I’ll try my best to improve the scripts.

Thank you.

Feel free to contact me on Twitter.

Troubleshooting and Upgrading AD FS Farms

, ,

Troubleshooting and Upgrading an AD FS farms is usually a straightforward and easy task. There are many blogs detailing the process from Server 2012R2 to Server 2016/2019. Here are the general steps for upgrading a farm.

  1. Setup up a new server and install the AD FS role.
  2. Add the server to the existing farm.
  3. Set the new server as the primary server.
  4. Point the other servers to the new primary server.
  5. Install additional servers to the AD FS farm.
  6. Uninstall AD FS on the old servers to remove them from the farm.

What are your options when this process doesn’t work? In my case, I could not add a new server to the farm. PowerShell and the GUI both returned errors when attempting to add a new Windows Server 2019 to the farm. I started looking at troubleshooting options and eventually decided to proceed with an attempt to in-place upgrade the Server 2012R2 farm to Server 2019.

Troubleshooting

Microsoft provides some powerful tools for troubleshooting AD FS issues. AD FS Help provides several troubleshooting guides and diagnostic tools that can help resolve issues with your AD FS farm. The tools are located here: https://adfshelp.microsoft.com/.

The AD FS Diagnostic Analyzer tool can provide a health check for you AD FS farm. To use the AD FS Diagnostic Analyzer, you need to install the AD FS Toolbox PowerShell Module.

Install-Module -Name ADFSToolbox -Force

Import-Module ADFSToolbox -Force

Once the AD FS Toolbox is installed, run the Export-AdfsDiagnosticsFile command which will generate a JSON file for upload. The command will run against the local AD FS server unless the farm is Windows Server 2016 or higher. You can also list the servers with the -adfsServers parameter. The -adfsServers parameter is required for 2012R2 farms. Upload the JSON file to the https://adfshelp.microsoft.com/ site and the site will display the Health Test Results. The site will detail any problems and offer step by step guides or links to documentation to remediate the issues.

Backup and Restore

There is also the AD FS Rapid Restore Tool found here: https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/ad-fs-rapid-restore-tool. The tool backs up the following AD FS configuration data:

  • AD FS database
  • Configuration file
  • Automatically generated token signing and decrypting certificates and private keys
  • SSL certificate and any externally enrolled certificates and corresponding private keys
  • The custom authentication providers, attribute stores, and local claims provider trusts that are installed

Unfortunately, the Restore Tool only supports a restore to the same version as the backup. This means that you cannot use this method to restore AD FS to a newer version of AD FS. You could restore the AD FS farm to a new set of servers and attempt to add upgraded servers to the farm as outlined above.

Other Tools

The https://adfshelp.microsoft.com/ site provides a number of other tools and support options. There is a full list of AD FS event items for 2012R2/2016/2019 with ID, Name, and Description. Claims X-Ray assists with debugging claims issues in your applications. The AD FS Event Module provides tools to gather and review the events from multiple servers. There are several more tools available as well.

In-Place Upgrade

Even after running the diagnostic tools several times and making the recommended changes, I still was unable to add a new server to the existing farm. The other option is to attempt an in-place upgrade of the servers in the AD FS farm. Technically, this is not supported as upgrading Windows Server with AD FS installed will uninstall the AD FS role. Prior to attempting this method, I made a snapshot (Hyper-V virtual Machines) of the AD FS servers and a backup of the AD FS farm’s current state. I started by upgrading the secondary AD FS server first. If I was unable to add the server back into the farm and promote it to the primary server, then my plan was to use the AD FS Restore Tool and rebuild the farm. These are the steps I took for the in-place upgrade.

  1. Upgrade the secondary AD FS server to Server 2019.
  2. Install the AD FS role.
  3. Add the upgraded server back into the farm.
  4. Set the server as the primary AD FS server in the farm.
  5. Verified that AD FS was still working for our services.
  6. Upgrade the former primary server, reinstall the AD FS role and set it as the primary server.

I also did an in-place upgrade of the Web Application Proxy server and added it to the farm with the Install-WebApplicationProxy cmdlet. The final step is raising the farm functional level with the Invoke-AdfsFarmBehaviorLevelRaise cmdlet. This will enable the new features to Verify the update completed with the Get-AdfsProperties | Select CurrentFarmBehavior cmdlet.

While it is not a supported option, the in-place upgrade of my AD FS farm worked perfectly. Hopefully, it is not an option you will need.

read more Troubleshooting and Upgrading AD FS Farms

The Cure For Your SSU Fever

, , , , , , , , ,