Challenges in Managing SSH Keys – and a Call for Solutions
Total Page:16
File Type:pdf, Size:1020Kb
Challenges in Managing SSH Keys – and a Call for Solutions Tatu Ylonen University of Helsinki [email protected] Abstract In some cases a single key can access tens of thousands of servers, such as for keys originally installed for emergency SSH (Secure Shell) uses public keys for authenticating servers response, auditing, or patch management purposes. Some of and users. While SSH’s key management design was great these keys are very old. for grass-roots deployments, it is now causing significant Second, extensive monitoring of the usage of the keys has operational and security challenges in large computing envi- revealed that typically about 90% of the authorized keys are ronments. This paper summarizes progress in SSH key man- unused. That is, they are access credentials that were provi- agement so far, highlights outstanding problems, and presents sioned years ago, the need for which ceased to exist or the requirements for a long-term solution. Proposals are solicited person having the private key left, and the authorized key from the research community to address the issue; so far the was never deprovisioned. Thus, the access was not terminated industry has been unable to come up with an ideal solution. when the need for it ceased to exist. This signifies a failure of The problem is of high practical importance, as most of our identify and access management (IAM) practices. critical Internet infrastructure, cloud services, and open source software development is protected using these keys. In many organizations – even very security-conscious or- ganizations – there are many times more obsolete authorized keys than they have employees. Worse, authorized keys gener- 1 Introduction ally grant command-line shell access, which in itself is often considered privileged. We have found that in many organiza- SSH (Secure Shell) [28,29] is an encryption protocol that is tions about 10% of the authorized keys grant root or adminis- used for managing most Unix/Linux servers, Internet routers, trator access. SSH keys never expire. and much of cloud computing infrastructure. It is also widely Third, it is rare for organizations to have the capability to used for committing code to version control repositories and quickly replace SSH user keys even when they know there inside various systems management tools. has been a breach and that private keys may have been com- The SSH protocol provides host authentication using public promised. keys associated with hosts, called host keys. These keys are The final issue relates to attitudes and operations. Many used to prevent man-in-the-middle attacks, such as stealing organizations do not want to know. If they acknowledged the passwords or command injection after authentication. issue, they would have to do something about it, because all The protocol also uses public keys for user authentication. the major security regulations require controlling who has These keys are called user keys. While there are several other access to what systems and data. Bringing SSH keys under authentication methods available, public key authentication is management is a substantial effort, and many companies kick the method of choice for automation. It is used, for example, the can down the road – even when there are literally mil- for accessing Linux instances in Amazon AWS, committing lions of unaccounted-for credentials granting access to their source code into github, for automating systems management, systems. and for single sign-on. We have recorded over five million There is an established process for managing SSH keys daily logins using public key authentication in some large [30]. However, the current process is based on managing financial institutions. user keys in authorized keys files and only addresses access We have made several alarming discoveries regarding user internal to an organization. One challenge is that SSH key keys. First is their sheer number: many large financial insti- based access is often used by automated processes, while most tutions have several million keys authorized to log in across IAM implementations only deal with access by people. Host tens of thousands, sometimes over a hundred thousand servers. key management is also not really addressed by the current 1 process. is not reliably serving its function of preventing man-in-the- There has been extensive research with hundreds of papers middle attacks. Thus the mechanism needs to be augmented on PKI (Public Key Infrastructure), but very little on SSH key or replaced by a mechanism that provides better security and and access management, even though it is arguably at least as smoother operations in large environments. critical for the core infrastructure. The contributions of this paper include: 2.2 Attempts to improve host key manage- • detailing the SSH key management problem and its im- ment portance • reviewing attempts to solve it so far, and The SSH protocol was originally designed to work without • providing requirements and design criteria for fully solv- any centralized key infrastructure, as none existed for ordinary ing it. servers in 1995, and even today one really only exists for HTTPS servers. Also, the size of large IT environments has grown since then by orders of magnitude. 2 Host keys Automatically collecting SSH keys and distributing them to system-wide known host files within an enterprise was The SSH client uses the server’s host key for authenticating attempted in the early 2000s with the Tectia Manager product. the server. In OpenSSH, for example, host keys of known The approach works reasonably well for up to a few thousand servers are stored in a known hosts file. When the user first servers, but thereafter raw copying of host keys becomes un- connects to a server, he/she is shown the fingerprint of the wieldy. First, the known hosts files are simple text files that server’s host key and asked whether to accept it. On later are linearly scanned for the matching host key. With tens of connections, OpenSSH verifies that the key has not changed, thousands of keys, the files grow to many megabytes, and the and asks the user to accept the new key if it has changed. Both cost of scanning through them on every login becomes pro- personal experience and available research [9] indicate that hibitively expensive. Furthermore, in a large environment of, users generally accept unknown host keys without verifying say, 100,000 servers, there are easily 2000 servers reinstalled them. every day, typically with their host keys changing. The cost and complexity of distributing the keys becomes rather high, 2.1 Problems with TOFU especially if regular key rotation is to be implemented (key rotation meaning changing the keys). Finally, many clients are SSH’s original and still dominant approach to host key man- laptops or mobile devices that are only sporadically connected agement is called TOFU (Trust on First Use) [31]. While this to the company network and whose client implementations approach was great for grass-roots deployment, it has become are not designed to work with very large numbers of known a problem. host keys. TOFU works well for small static environments with tech- Peng and Zhao [21] describes a method for initializing savvy users, but offers little security in large dynamic envi- trust for network devices using SSH, where a central system ronments. This has effectively led to a breakdown of SSH’s manages and generates a list of known hosts and distributes it man-in-the-middle attack protection and opened it up for vari- to the managed devices. However, in their system a human is ous active network-level attacks, including password stealing, assumed to verify the fingerprint of the central device. data theft, and command injection. Numerous free tools are Another attempted approach has been to store SSH host available for performing these attacks. keys in DNS (Domain Name System) using SSHFP records To illustrate the problem, we have seen a very tech-savvy [24]. Even though this is an IETF standards-track specifica- organization with 30,000 general-purpose hosts sharing the tion that has been around for over a decade, it has not seen same host key. In Heninger et al [11], 65% of all host keys extensive use. The approach relies on DNSSEC [3] for secu- found on the public Internet were used on more than one host rity, but DNSSEC is not yet very widely deployed in most – many of them network devices such as routers, firewalls, and countries. Also, due to the way DNS update permissions are server management cards. structured in most organizations and lack of automation, up- In practice users do not check host keys [9]. Users do not dating fingerprints in DNS is often too cumbersome. understand the warnings about changed host keys and even for Jones [13] presents a layered approach that uses DNSSEC experts, verifying the keys is too cumbersome to do reliably. to store the location and public key of a key distribution server, Furthermore, in a large enterprise hundreds if not thousands but keys for individual entities are stored in the key distribu- of new hosts are installed or reinstalled every day, leading to tion server(s). This securely moves individual host keys out significant key churn. from DNS, which is administratively much simpler to im- The uniqueness, security, and validation of host keys is plement than SSHFP. Similar mechanisms could also enable critical for man-in-the-middle attack prevention. We thus find automatic discovery of key management servers. It might be ourselves in a situation where the host authentication in SSH possible to extend this system for distributing authorized keys 2 as well. connections when the host key does not match a known key. Redhat Linux supports using the ProxyCommand feature in However, strictly enforcing host keys can prevent access to the OpenSSH client to fetch the host key for the host being servers and break business processes when a host key changes connected to into a known hosts file, and reading it from that due to a reinstallation, and forcing key acceptance on users file.