SSL Handshake Troubleshooting

About

This article is meant to provide locations to look when dealing with handshake errors. Starting with the Ciphers used in the SSL handshake and what can happen if they do not match. This is applicable to Venafi due to our connections to the CA's that may be outside your network and on servers you do not manage.

The server may have settings that you do not have control over and the server Venafi is installed on needs to be modified to allow communication between the two servers.

Alternatively if the Ciphers are not causing the issue, a certificate error can also be causing the issue, things in between your server and the remote server such as Load Balancers, or server requirements for a client certificate.

Protocols

Protocols are used in an SSL handshake in the Client Hello and Server Hello when establishing TCP connections between servers.

Example Protocols: (Tls1.0, Tls1.1, Tls1.2, Ssl3)

The Protocols can be found in the following Registy location

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols

Each protocol will have two settings for each, the Server and the Client. The client will handle the outgoing connections, while the Server will handle the incoming connections

 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Client
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Server

By default, no attribute or values will be specified, for example the only one my system has listed is SSL 2.0.

To Disable SSL 3.0 you would add the Entry for SSL 3.0 under Protocol, then adding the Client and Server container under any Protocols you wish to modify. Below will show what SSL 3.0 would look like:

 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Client

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Server

One last step to disable the ciphers. Add a DWORD attribute called "DisabledByDefault" Right click "DisabledByDefault" and then select Modify, Under Value data it will accept one of the following values:

Attribute Name	Value Data: (Enabled)
DisabledByDefault	0x00000000

	Value Data: (Disabled)
DisabledByDefault	0x00000001

This process is the same for any cipher you need to disable. Alternatively if you need to enable the cipher, changing the value to 0x00000000

In the Image below it shows the default for Windows Server 2012 R2 Standard. The entries for the other protocols are enabled, because the value DisabledByDefault is not present, which is the equivalent of 0 in this case.

 

Following the thought process logically:

For Disabled:

1. If the value is 0x00000001
2. "DisabledByDefault is True"
3. Which means it is disabled currently

For Enabled:

1. If the value is 0x00000000
2. "DisabledByDefault is False"
3. Which means it is enabled and will be sent along in a SSL handshake

Additionally the ciphers can be affected in the network in the following way:

1. Traffic goes through Load Balancer/Proxy or edge device
2. The first handshake will be between the requesting device and the remote device
3. The handshake can be successful here and fail when the device tries to establish the connection to the device originally requested
4. This is because on select devices you can set up ciphers for establishing the connections, these will be used in the handshake

You may make a solid connection to the load balancer or other device, when this device attempts to make a connection to the server originally requested, the device and the server can then have a handshake error. This is not easy to get from context, so here is a table that will attempt to show this:

	Client	Device	Server
Protocols Enabled	Tls1.0,Tls1.1,Tls1.2	Tls1.0	Tls1.2

In the example above, you can establish a connection between the client and the device, the server will fail. This will cause Venafi to fail when attempting to reach a Certificate Authority that only has Tls1.2 enabled.

Here is the settings to resolve this:

	Client	Device	Server
Protocols Enabled	Tls1.0,Tls1.1,Tls1.2	Tls1.0,Tls1.2	Tls1.2

Certificate Errors

Any of the following scenarios can cause certificate errors:

1. Remote certificate expired
2. Remote certificate not trusted
3. Remote server requires a client certificate
4. Client certificate expired 
5. Wrong client certificate provided

Notably 2,3,4,5 are the ones that are not distinctively clear. These are the things to look for when dealing with communication issues after the ciphers have been investigated

#2 would indicate that the certificate is either self signed or signed by an internal CA that is not trusted on the requesting machine

#3 In the Venafi product it may not be inherently clear that this is what is causing a communication failure. Because of this you will want to ensure that if the remote server requires a user certificate

#4 If a CA is not providing the client certificate for you (Internal CA for example) you can check the validity of the certificate if it has been uploaded to Venafi, and then import the new certificate. This can be done on the same credential object in Venafi if needed. This ensures any location that needs to use this certificate is pointing back to the same location and an update will not be needed to those objects.

#5 Often this is caused by the CA providing a new certificate to be used with requests to them, the old certificate will then be rejected on new requests