Cylance, PKI, and You

, ,

Maintaining PKI functionality through the upgrade to CylancePROTECT 1580

We recently deployed Cylance 1584 across our environment. Cylance implemented version 2 of their Memory and Script Controls with version 1580, and we were a few versions behind. We were expecting a few bumps in the road as a result, but none as significant as the loss of our PKI environment.

How it started

The button was clicked, and Cylance 1584 was deployed. Tickets were coming in from our SIEM, but that that was to be expected. The first alert came in the next day. One of the support engineers reported that devices were failing to enroll into Intune through AutoPilot. The process failed at the Certificate section of the Device Setup. She was able to continue AutoPilot to get to the desktop, but WiFi and VPN weren’t connecting. This meant the device had no device certificate from PKI.

Troubleshooting PKI

Event Viewer is always a good place to start for troubleshooting. There were a lot of error events in the Application log. These events had ID 29 with Source NetworkDeviceEnrollmentService. The message read The password in the certificate request cannot be verified. It may have been used already Obtain a new password to submit this request. Google had no definite answers, but there were more logs to review.

The next logs I found always came in pairs.

Event IDSourceMessage
1000Application ErrorFaulting Application Name: w3wp.exe
1023.NET RuntimeApplication: w3wp.exe

These logs show that this w3wp.exe executable is being blocked. The time stamps on the events line up with the Cylance update, so it seems obvious that Cylance is blocking the executable. Oddly enough, there are no alerts in Cylance that this executable is blocked. This w3wp.exe process seems to be the key to the certificate issue, but there isn’t enough information in the logs to fix the issue.

The next log to investigate is the System log. Searching the logs around the same timestamp as the w3wp.exe logs, I find the next set of logs.

Event IDSourceLevelMessage
5011WASWarningA process ervicing application pool 'Microsoft Intune CRP Service Pool' suffered a fatal communication error with the Windows Process Activation Service. The process id was 'xxxxx'. The data field contains the error number.
5002WASErrorApplication pool 'Microsoft Intune CRP Service Pool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.

These logs indicate that the w3wp.exe executable has failed enough times that the IIS Application Pool has been disabled.

Putting the pieces together

Cylance 1580 will occasionally block executables that depend on .NET runtimes. When this happens, you will not get alerts about the executable in the Admin Console. In the policy that you deploy to your web servers, you will need to include the following exclusions to Memory Actions:

  • \Windows\System32\inetsrv\w3wp.exe
  • \Windows\sysWOW64\inetsrv\w3wp.exe
  • \Program Files (x86)\IIS Express\iisexpress.exe

It is worth noting that at this time, Cylance can apply a hotfix to your account that will automatically apply those exclusions. This will upgrade your version to 1584.46. This will also be included in version 3 whenever that is released later this year.

You should also keep this in mind for troubleshooting additional applications in your environment. By quickly filtering your Application log by event ID 1000, you can gauge if there are any blocks on your system caused by Cylance. Not all ID 1000 events will be caused by Cylance 1580, but it can be a good starting point for troubleshooting.

Intune – Autopilot Hash Generation Factory

, , , , , , ,

I’ve been doing more and more work with Intune lately and have been looking at different methods of Autopiloting devices.  My go-to has been OSDCloud by David Segura. It is a great tool for quickly getting devices from many different vendors up and running with Autopilot.  Definitely check it out.

I ran into a scenario where I needed to mimic the experience not of the end user, but of IT managing devices enrolled by the vendor prior to them ever appearing in Intune.  This is “easy”, albeit a bit time consuming.  You have to get a full Windows OS onto the device, log in and run Michael Niehaus’ Get-WindowsAutopilotInfo script.  You then import the generated CSV into Intune.  Finally you rebuild the device with a factory install of Windows 10 or11 and you can then Autopilot it into Intune.

That’s fine for one or two devices, but I needed to get a number of VMs staged up as if they were ordered from the vendor.  I needed to automate the grunt work….

Read more