Error: "Failed to unwrap inner PKCS7" when doing SCEP requests

Symptom:

SCEP requests to Venafi are failing. The Default SQL Channel logs show following:

Network Device Enrollment - Failed to parse CSR	Failed to extract the subject from the received CSR
Network Device Enrollment - Failed to unwrap inner PKCS7	Failed to properly decode the inner PKCS#7 envelope in the received SCEP data

 

Cause:

1. Typically this is caused by the payload sent to Venafi SCEP being encrypted to the wrong certificate as configured in the RA Certificate settings in Venafi.

2. This may also be caused by Key Usage being set to "Digital Signature"

Resolution:

Item 1 Resolution:

Ensure that the certificate settings on the system sending the SCEP request matches those configured in the RA settings within Venafi (see graphic below).

Item 2 Resolution:

Modify the Key Usage to:  "Digital Signature, Key Encipherment" and try again.  Make sure to reset the VEDSCEP App pool in IIS after making any changes.

More info:

Example on how to configure Venafi SCEP:

https://support.venafi.com/hc/en-us/articles/215914547

How to determine what cert is being used in the request:

(NOTE: This will only work if the payload plus IIS log data is less than 4096 bytes, which is the maximum length for a single IIS log entry. Otherwise, the payload will be truncated.)

 

1. Extract the payload from IIS logs

2. Use following Powershell script to decode the URL formatted payload

   [Reflection.Assembly]::LoadWithPartialName("System.Web") | Out-Null
   $encode = '****place encoded payload data here***'
   $b64 = [System.Web.HttpUtility]::UrlDecode($Encode)
   $filename = 'C:\temp\filetolookat.p7b'
   $bytes = [Convert]::FromBase64String($b64)
   [IO.File]::WriteAllBytes($filename, $bytes)

3. 3. View the file with certutil command:

   certutil.exe -dump $filename

4. Find the "Recipient Info" section:

   Recipient Info[0]:
   CMSG_KEY_TRANS_RECIPIENT(1)
   CERT_ID_ISSUER_SERIAL_NUMBER(1)
   Serial Number: 187fdfgg2
   Issuer: CN=Venafi Root CA

5. Compare the Serial Number of the certificate used to encrypt the payload to what is configured in Venafi SCEP RA settings: