Info: Configuring SSH server and logging system for SSH key usage tracking – Venafi Customer Support

Applies To:

TPP 15.3 and newer.

Summary:

Starting with the Venafi TPP 15.3 release, Venafi SSH Key Manager, which is part of Venafi Trust Protection Platform, provides capabilities to track usage of SSH keys in your environment. TPP then matches key usage data with existing key inventory data and helps identify the following issues with SSH keys:

Keys that have not been used for certain amount of days and therefore should be cleaned up
Known authorized keys that have been used from unknown client (there is no known private key on source IP address)
Keys being used but not tracked in Venafi SSH Key Manager

In order to enable collecting of SSH key usage data, some amount of configuration is needed for the SSH server and logging system. This article describes the required steps with some examples for most common software. It's important to note that key usage logs can only be sent to an agent that is installed on Linux.

More Information

A typical configuration of SSH logging for the enterprise looks like this:

SSH server puts its logs into a local syslog system, which in turn forwards the logs to a centralized syslog or SIEM system for log auditing and analytics.

SSH key usage tracking requires the Venafi Agent be installed on a machine with logging information (be it a local or centralized log application). As of TPP 15.3, SSH Key Usage work needs to be configured for the agent. Please refer to Venafi TPP documentation for more details on configuring SSH Key Usage work.

Upon receiving the work, the agent starts listening on TCP port 689 for key usage data and forwards any data it "hears" on that port to TPP on the configured key usage work schedule.

For best security, TCP packets are only allowed from localhost, so the agent must be installed on the machine that processes logs. And, the logging system needs to be configured to forward SSH key usage logs to port 689 on localhost. A typical setup will look like this:

Configuring SSH server to output key fingerprints in log

The following sections contain steps to configure SSH server and the logging system for this setup.

As of 15.3, TPP supports OpenSSH format of logs. By default, OpenSSH 6.2 and older does not do that, VERBOSE log level needs to be enabled. OpenSSH 6.3 and newer have fingerprint logged with INFO level.

For OpenSSH servers version 6.2 and older:

1. Set LogLevel directive value in sshd_config to VERBOSE (default value is INFO)

LogLevel VERBOSE

2. Restart SSH server

Configuring the logging system to send fingerprints to TCP port 689

Logging system needs to be configured to forward logs from the sshd process to local port 689. The following subsections contain information on setting this up for a few common logging systems. For other logging systems, refer to their documentation on TCP forwarding.

Note: forwarding logging to port 689 might result in overfilling logs with error messages if Venafi Agent is not running or cannot start socket listener for some reason. If you experience this, recommended way is to filter out connection failed messages to port 689.

rsyslog

1. Add forwarding rule to rsyslog configuration (typically /etc/rsyslog.conf)

if $programname == 'sshd' then @@127.0.0.1:689

[optional] You can also set up reliable forwarding with rsyslog.

2. [recommended] Setup rsyslog to print fully qualified domain names, if it's not already the case. This will help TPP properly identify machine in use. Add the following to rsyslog configuration

$PreserveFQDN on

3. Restart rsyslog

service rsyslogd restart

Some Linux distributions (RHEL, CentOS) are not able to have rsyslog redirect traffic to ports other than 514 (default syslog port) by SELinux. To enable key usage collection, rsyslog will need to be enhanced to change the port, for example:

yum install policycoreutils-python.x86_64
semanage port -a -t syslogd_port_t -p tcp 689

syslog-ng

1. Add forwarding rule to syslog-ng configuration (typically /etc/syslog-ng/syslog-ng.conf)

destination d_ssh_vagent { tcp("127.0.0.1" port(689)) log_fifo_size(1000); };

filter f_ssh { program("sshd"); };

log { source(s_src); filter(f_ssh); destination(d_ssh_vagent); };

2. [recommended] Setup syslog-ng to print fully qualified domain names, if it's not already the case. This will help TPP properly identify machine in use. Make sure the corresponding option is on in syslog-ng configuration

options {
  ..
  use_fqdn(yes);
}

3. Restart syslog-ng

service syslog-ng restart

Splunk

Setup forwarding syslog-formatted messages to third-party system:

1. Enable SplunkForwarder app:

splunk enable app SplunkForwarder -auth user:password

2. Edit <SPLUNK_HOME>/etc/apps/SplunkForwarder/local/outputs.conf

[tcpout]

[syslog]

[syslog:venafi]
server = 127.0.0.1:689
type = tcp

3. Edit <SPLUNK_HOME>/etc/apps/SplunkForwarder/local/props.conf

[source::*]
TRANSFORMS-routing = venafiOut

4. Edit <SPLUNK_HOME>/etc/apps/SplunkForwarder/local/transforms.conf

[venafiOut]
REGEX = sshd
DEST_KEY = _SYSLOG_ROUTING
FORMAT = venafi

5. Restart Splunk

Troubleshooting:

If the key usage data is not appearing in TPP, check the following:

1. Venafi Agent is running on log machine.

2. SSH Key Usage work is configured for that agent via group it is part of, and that it receives it. Agent should print that it received VED Agent Log Deliver when run with debug log level.

3. OpenSSH is outputting fingerprints. This can be checked by looking for "Accepted publickey" and "Found matching" strings in OpenSSH logs.

4. Logging system correctly forwards logs. Check your logging system configuration, it's also helpful to temporarily stop agent and set up debug port listener for testing, for example nmap:

nc –lk –p 689

When agent receives its first log strings, it will load recognizers from TPP which will be reflected by a corresponding log message.