This article is meant to provide locations to look when dealing with handshake errors. Starting with the Ciphers used in the SSL handshake and what can happen if they do not match. This is applicable to Venafi due to our connections to the CA's that may be outside your network and on servers you do not manage.
The server may have settings that you do not have control over and the server Venafi is installed on needs to be modified to allow communication between the two servers.
Alternatively if the Ciphers are not causing the issue, a certificate error can also be causing the issue, things in between your server and the remote server such as Load Balancers, or server requirements for a client certificate.
Protocols are used in an SSL handshake in the Client Hello and Server Hello when establishing TCP connections between servers.
Example Protocols: (Tls1.0, Tls1.1, Tls1.2, Ssl3)
The Protocols can be found in the following Registy location
Each protocol will have two settings for each, the Server and the Client. The client will handle the outgoing connections, while the Server will handle the incoming connections
By default, no attribute or values will be specified, for example the only one my system has listed is SSL 2.0.
To Disable SSL 3.0 you would add the Entry for SSL 3.0 under Protocol, then adding the Client and Server container under any Protocols you wish to modify. Below will show what SSL 3.0 would look like:
One last step to disable the ciphers. Add a DWORD attribute called "DisabledByDefault" Right click "DisabledByDefault" and then select Modify, Under Value data it will accept one of the following values:
|Attribute Name||Value Data: (Enabled)|
|Value Data: (Disabled)|
This process is the same for any cipher you need to disable. Alternatively if you need to enable the cipher, changing the value to 0x00000000
In the Image below it shows the default for Windows Server 2012 R2 Standard. The entries for the other protocols are enabled, because the value DisabledByDefault is not present, which is the equivalent of 0 in this case.
Following the thought process logically:
- If the value is 0x00000001
- "DisabledByDefault is True"
- Which means it is disabled currently
- If the value is 0x00000000
- "DisabledByDefault is False"
- Which means it is enabled and will be sent along in a SSL handshake
Additionally the ciphers can be affected in the network in the following way:
- Traffic goes through Load Balancer/Proxy or edge device
- The first handshake will be between the requesting device and the remote device
- The handshake can be successful here and fail when the device tries to establish the connection to the device originally requested
- This is because on select devices you can set up ciphers for establishing the connections, these will be used in the handshake
You may make a solid connection to the load balancer or other device, when this device attempts to make a connection to the server originally requested, the device and the server can then have a handshake error. This is not easy to get from context, so here is a table that will attempt to show this:
In the example above, you can establish a connection between the client and the device, the server will fail. This will cause Venafi to fail when attempting to reach a Certificate Authority that only has Tls1.2 enabled.
Here is the settings to resolve this:
Any of the following scenarios can cause certificate errors:
- Remote certificate expired
- Remote certificate not trusted
- Remote server requires a client certificate
- Client certificate expired
- Wrong client certificate provided
Notably 2,3,4,5 are the ones that are not distinctively clear. These are the things to look for when dealing with communication issues after the ciphers have been investigated
#2 would indicate that the certificate is either self signed or signed by an internal CA that is not trusted on the requesting machine
#3 In the Venafi product it may not be inherently clear that this is what is causing a communication failure. Because of this you will want to ensure that if the remote server requires a user certificate
#4 If a CA is not providing the client certificate for you (Internal CA for example) you can check the validity of the certificate if it has been uploaded to Venafi, and then import the new certificate. This can be done on the same credential object in Venafi if needed. This ensures any location that needs to use this certificate is pointing back to the same location and an update will not be needed to those objects.
#5 Often this is caused by the CA providing a new certificate to be used with requests to them, the old certificate will then be rejected on new requests