Cylance, PKI, and You

, ,

Maintaining PKI functionality through the upgrade to CylancePROTECT 1580

We recently deployed Cylance 1584 across our environment. Cylance implemented version 2 of their Memory and Script Controls with version 1580, and we were a few versions behind. We were expecting a few bumps in the road as a result, but none as significant as the loss of our PKI environment.

How it started

The button was clicked, and Cylance 1584 was deployed. Tickets were coming in from our SIEM, but that that was to be expected. The first alert came in the next day. One of the support engineers reported that devices were failing to enroll into Intune through AutoPilot. The process failed at the Certificate section of the Device Setup. She was able to continue AutoPilot to get to the desktop, but WiFi and VPN weren’t connecting. This meant the device had no device certificate from PKI.

Troubleshooting PKI

Event Viewer is always a good place to start for troubleshooting. There were a lot of error events in the Application log. These events had ID 29 with Source NetworkDeviceEnrollmentService. The message read The password in the certificate request cannot be verified. It may have been used already Obtain a new password to submit this request. Google had no definite answers, but there were more logs to review.

The next logs I found always came in pairs.

Event IDSourceMessage
1000Application ErrorFaulting Application Name: w3wp.exe
1023.NET RuntimeApplication: w3wp.exe

These logs show that this w3wp.exe executable is being blocked. The time stamps on the events line up with the Cylance update, so it seems obvious that Cylance is blocking the executable. Oddly enough, there are no alerts in Cylance that this executable is blocked. This w3wp.exe process seems to be the key to the certificate issue, but there isn’t enough information in the logs to fix the issue.

The next log to investigate is the System log. Searching the logs around the same timestamp as the w3wp.exe logs, I find the next set of logs.

Event IDSourceLevelMessage
5011WASWarningA process ervicing application pool 'Microsoft Intune CRP Service Pool' suffered a fatal communication error with the Windows Process Activation Service. The process id was 'xxxxx'. The data field contains the error number.
5002WASErrorApplication pool 'Microsoft Intune CRP Service Pool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.

These logs indicate that the w3wp.exe executable has failed enough times that the IIS Application Pool has been disabled.

Putting the pieces together

Cylance 1580 will occasionally block executables that depend on .NET runtimes. When this happens, you will not get alerts about the executable in the Admin Console. In the policy that you deploy to your web servers, you will need to include the following exclusions to Memory Actions:

  • \Windows\System32\inetsrv\w3wp.exe
  • \Windows\sysWOW64\inetsrv\w3wp.exe
  • \Program Files (x86)\IIS Express\iisexpress.exe

It is worth noting that at this time, Cylance can apply a hotfix to your account that will automatically apply those exclusions. This will upgrade your version to 1584.46. This will also be included in version 3 whenever that is released later this year.

You should also keep this in mind for troubleshooting additional applications in your environment. By quickly filtering your Application log by event ID 1000, you can gauge if there are any blocks on your system caused by Cylance. Not all ID 1000 events will be caused by Cylance 1580, but it can be a good starting point for troubleshooting.

Potential USMT Errors & Resolution

, , , , , , , , ,
Potential USMT Errors


Failure when “trying to reboot into WinPE”
appears on the Task Sequence UI (forgot to take screenshot)

 Potential USMT Errors & Resolution

 

      Check the “SMSTS.LOG” “C:WindowsCCMLogs”

 

         In the screenshot below look for a line like “Unable to find a volume that is suitable for staging the boot image” in the log file 

 

Potential USMT Errors
       This is typically for 2 different reasons.
      1.  The drive is locked by bitlocker encryption and you need to first disable bitlocker in the task            sequence. once this is done then the task sequence engine can identify the drive to stage the        boot image locally
      2.   This can also be because the drive is still undergoing the encryption process. You must wait             until the drive is fully encrypted before you can execute this task sequence from windows.
       To verify the drive is finished encrypting launch powershell as an admin and type “Get-BitlockerVolume” and do not attempt to run the installation until the “Encryption KeyProtector Percentage” is are 100%  
Potential USMT Errors
NOTE: it is possible you receive this error when you are trying to stage the boot image onto the disk and the disk cannot be read b/c of a different encryption software locks the disk. For example if you are running Dell Credant you must be logged into the system then the TS Engine will be able to read the disk.
To identify Dell Credant systems via SQL…this is a hybrid of one my queries that I use in my production environment but you can modify it to abosrb only bitlocker information by commenting out not needed parts

SELECT Distinct

v_R_System.Name0 AS System,

Computer_System_DATA.Model00 AS [System Model],

___System_INSTALLED_SOFTWARE0.ARPDisplayName00,

V_R_System.AD_Site_Name0 AS [AD Site],

CASE V_R_System.Build01

When ‘6.1.7601’ THEN ‘Windows 7’

WHEN ‘10.0.14393’ THEN ‘Win 10 v1607’

WHEN ‘10.0.15063’ THEN ‘Win 10 v1703’

END AS [Operating System],

CASE V_R_System.Client0

When ‘0’ THEN ‘No Client’

WHEN ‘1’ THEN ‘Client Installed’

END AS [Client],

v_GS_ENCRYPTABLE_VOLUME.DriveLetter0 AS [Drive Letter],

–v_GS_ENCRYPTABLE_VOLUME.ProtectionStatus0 AS [Protection Status],

CASE v_GS_ENCRYPTABLE_VOLUME.ProtectionStatus0

WHEN ‘0’ THEN ‘not encrypted’

WHEN ‘1’ THEN ‘encrypted’

WHEN ‘2’ THEN ‘Encrypted Requires Pin’

END AS [Bitlocker Status]

FROM

v_GS_ENCRYPTABLE_VOLUME

INNER JOIN v_R_System ON v_GS_ENCRYPTABLE_VOLUME.ResourceID = v_R_System.ResourceID

INNER JOIN Computer_System_Data ON V_R_System.Name0 = Computer_System_Data.Name00

INNER JOIN INSTALLED_SOFTWARE_DATA AS ___System_INSTALLED_SOFTWARE0 ON ___System_INSTALLED_SOFTWARE0.MachineID = V_R_System.ResourceID

Where

v_GS_ENCRYPTABLE_VOLUME.DriveLetter0 = ‘C:’

AND Computer_System_DATA.Model00!= ‘VMware Virtual Platform’

AND Computer_System_DATA.Model00!= ‘Virtual Machine’

–AND v_GS_ENCRYPTABLE_VOLUME.ProtectionStatus0 = ‘0’ –for not bitlocker encrypted systems–

–AND V_R_System.Name0 = ‘P620268’

AND ___System_INSTALLED_SOFTWARE0.ARPDisplayName00 like N’Credant_WindowsShield%’

 

Potential USMT Errors

 

 
Failure when trying to connect to SMP Share
 
This failure is more often seen when trying to rerun on a failed system (can be seen during backup or restore part of the process) 
 
Potential USMT Errors
For this we corrected the issue is to open PowerShell and running the following.

 

Remove-Item -Path ‘HKLM:SOFTWAREMicrosoftSystemCertificatesSMSCertificates*’ -force; restart-service ccmexec

I would also recommend opening registry location to verify this has successfully been deleted.
Once this runs successfully you should then be able to re-run the task sequence successfully.
File Not Found: 
of course there is the standard make sure your commands are typed correctly. In the example below we see a file not found error. Make sure you type out your file names correctly etc.
Potential USMT Errors
Potential USMT Errors
NOTE: I have seen cases where a variable is set for Packages, but it does not always translate, so I tend to just hard code the package ID when I set restore/capture options into a variable see the example below
Potential USMT Errors
Connection to SMP Refused: 
Make sure when you being your deployment strategy that you plan for an adequate number of connections to the SMP. The default for this I believe is 100 connections, but that does not mean concurrent connections. The criteria that goes into the count is any established connection (completed or in progress) within the your deletion policy time period. If you have a problem where the connection is actively refused by the SMP you should increase the max allowed connections you have configured.
Potential USMT Errors
I tried to create as many Potential USMT Errors as I could think of to help out the community. If I  encounter anymore or can think of new ones I will add them to this blogpost.

SCCM Revoked Clients Registration

, ,

SCCM Revoked Clients Registration

I ran into an issue where a few sites would call my SCCM team indicating they were having client problems. They would say the clients do not have all action items and it has been over 2 hours since the system finished the OSD Process. I had a few initial thoughts but they all were wrong. First we verified that VMware tools on the MP was fine, boundaries were correct, and that the client was not stuck in provisioning mode. The next course of action was to connect to a machine, investigate the bad client and start checking log files.

The SCCM Client looked as below with certificate set to none as you can see below instead of self-signed. This points out the client has not yet registered with the MP.

SCCM Revoked Clients Registration

The next was to investigate the ClientIDManagerStartup.log which showed the error “Server Rejected registration Request: 3” This gave me the idea of clearing out the certs and trying to reinstall the client again.

SCCM Revoked Clients Registration

however this produced another GUID with the same error problem so the pointed me to check log files on the MP.

SCCM Revoked Clients Registration

This was quickly becoming a high exposure problem was our environment images anywhere between 25 – 175 machines a day. After a quick google search we were able to find a blog post where the solution was a few simple SQL lines.

The line below will identify systems that have revoked clients

Select * from ClientKeyData where isrevoked=1

SCCM Revoked Clients Registration

The line below will clear out the problematic requests. Once these are cleared the systems should be able to successfully register

Update ClientKeyData set isrevoked=0 where isrevoked=1

SCCM Revoked Clients Registration

Thanks to Emmanuel Rached blogpost below this was quickly resolved. The log files were screencaps from his blog post and everything else was from my environment. Please check out his blog as there is tons of other great stuff. https://www.emmanuelrached.com/2014/09/08/sccm-revoked-clients-registration/

ALSO CHECK: SQL Query to identify Win 10 security features for HP / DELL