Follow

Info: Configuring SSH server and logging system for SSH key usage tracking

Applies To:

TPP 15.3 and newer.

Summary:

 
Starting with 15.3 release, Venafi SSH Key Manager which is part of Venafi Trust Protection Platform provides capabilities to track usage of SSH keys in your environment. TPP then matches key usage data with existing key inventory data and helps identify the following issues with SSH keys:
  • Keys that have not been used for certain amount of days and therefore should be cleaned up
  • Known authorized keys that have been used from unknown client (there is no known private key on source IP address)
  • Keys being used but not tracked in Venafi SSH Key Manager
 
In order to enable collecting of SSH key usage data, some amount of configuration is needed for SSH server and logging system. This article describes required steps with some examples for most common software.

 

Typical configuration of SSH logging for enterprise looks like this:
syslog_config.png
 
SSH server puts its logs into local syslog system which in turn forwards the logs to centralized syslog or SIEM system for log audit and analytics.

 

SSH key usage tracking requires Venafi Agent installed on machine with logging information (be it local or centralized log application). As of 15.3, Linux platformIn TPP, SSH Key Usage work needs to be configured for agent. Please refer to Venafi TPP documentation for more details on configuring SSH Key Usage work.
Upon receiving the work, agent starts listening on TCP port 689 for key usage data and then forwards that data to TPP according configured key usage work schedule. For best security, TCP packets are only allowed from localhost, so agent has to be installed on the machine that processes logs. Logging system needs to be configured to forward SSH key usage logs to port 689 on localhost. Typical setup will look like this:
syslog_config_v.png
 
The following sections contain steps to configure SSH server and logging system for this setup.

Configuring SSH server to output key fingerprints in log

As of 15.3, TPP supports OpenSSH format of logs. By default, OpenSSH 6.2 and older does not do that, VERBOSE log level needs to be enabled. OpenSSH 6.3 and newer have fingerprint logged with INFO level.

For OpenSSH servers version 6.2 and older:

1. Set LogLevel directive value in sshd_config to VERBOSE (default value is INFO)

LogLevel VERBOSE

2. Restart SSH server

Configuring logging system to send fingerprints to TCP port 689

Logging system needs to be configured to forward logs from sshd process to local port 689. The following subsections contain information on setting this up for few common logging systems. For other logging systems, refer to their documentation on TCP forwarding.

Note: forwarding logging to port 689 might result in overfilling logs with error messages if Venafi Agent is not running or cannot start socket listener for some reason. If you experience this, recommended way is to filter out connection failed messages to port 689.

rsyslog

1. Add forwarding rule to rsyslog configuration (typically /etc/rsyslog.conf)

if $programname == 'sshd' then @@127.0.0.1:689

[optional] You can also set up reliable forwarding with rsyslog.

2. [recommended] Setup rsyslog to print fully qualified domain names, if it's not already the case. This will help TPP properly identify machine in use. Add the following to rsyslog configuration

$PreserveFQDN on

3. Restart rsyslog

service rsyslogd restart

 

Some Linux distributions (RHEL, CentOS) are not able to have rsyslog redirect traffic to ports other than 514 (default syslog port) by SELinux. To enable key usage collection, rsyslog will need to be enhanced to change the port, for example:

yum install policycoreutils-python.x86_64
semanage port -a -t syslogd_port_t -p tcp 689

syslog-ng

1. Add forwarding rule to syslog-ng configuration (typically /etc/syslog-ng/syslog-ng.conf)

destination d_ssh_vagent { tcp("127.0.0.1" port(689)) log_fifo_size(1000); };

filter f_ssh { program("sshd"); };

log { source(s_src); filter(f_ssh); destination(d_ssh_vagent); };

2. [recommended] Setup syslog-ng to print fully qualified domain names, if it's not already the case. This will help TPP properly identify machine in use. Make sure the corresponding option is on in syslog-ng configuration

options {
..
use_fqdn
(yes);
}

3. Restart syslog-ng

service syslog-ng restart

Splunk

Setup forwarding syslog-formatted messages to third-party system:

1. Enable SplunkForwarder app:

splunk enable app SplunkForwarder -auth user:password

2. Edit <SPLUNK_HOME>/etc/apps/SplunkForwarder/local/outputs.conf

[tcpout]

[syslog]

[syslog:venafi]
server = 127.0.0.1:689
type = tcp

3. Edit <SPLUNK_HOME>/etc/apps/SplunkForwarder/local/props.conf

[source::*]
TRANSFORMS-routing = venafiOut

 4. Edit <SPLUNK_HOME>/etc/apps/SplunkForwarder/local/transforms.conf

[venafiOut]
REGEX = sshd
DEST_KEY = _SYSLOG_ROUTING
FORMAT = venafi

5. Restart Splunk

 

More Info:

Troubleshooting

If the key usage data is not appearing in TPP, check the following:

1. Venafi Agent is running on log machine.

2. SSH Key Usage work is configured for that agent via group it is part of, and that it receives it. Agent should print that it received VED Agent Log Deliver when run with debug log level.

3. OpenSSH is outputting fingerprints. This can be checked by looking for "Accepted publickey" and "Found matching" strings in OpenSSH logs.

4. Logging system correctly forwards logs. Check your logging system configuration, it's also helpful to temporarily stop agent and set up debug port listener for testing, for example nmap:

nc –lk –p 689

When agent receives its first log strings, it will load recognizers from TPP which will be reflected by corresponding log message. 

Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk