Security of Communication Protocols and Services

87-30-01

DATA SECURITY MANAGEMENT SECURITY OF COMMUNICATION PROTOCOLS AND SERVICES

William Hugh Murray

INSIDE Protocols; Security Protocols; Secure Socket Layer (SSL); Secure-HTTP; Secure File Transfer Protocol (S-FTP); Secure Electronic Transaction (SET); Point-to-Point Tunneling Protocol (PPTP); Layer 2 Forwarding (L2F); Layer 2 Tunneling Protocol (L2TP); Secure Internet Protocol (Secure-IP or IPsec); Internet Security Association Key Management Protocol (ISAKMP); Password Authentication Protocol (PAP); Challenge Handshake Authentication Protocol (CHAP); Services; Telnet; File Transfer; Secure Shell (SSH 2)

INTRODUCTION For the last century, people have trusted the dial-switched voice-analog network. It was operated by one of the most trusted enterprises in the history of the world. It was connection-switched and point-to-point. While there was some eavesdropping, most of it was initiated by law en- forcement and was, for the most part, legitimate. While a few people carefully considered what they would say, most used the telephone automatically, without worrying about being overheard. Similarly, people were able to recognize most of the people who called, they trusted the millions of copies of the printed directories, and they trusted the network to connect only to the number dialed. While it is not completely justiﬁed, much of that automatic trust has been transferred to the modern digi- PAYOFF IDEA tal network and even to the Internet. The information security manager is confronted All other things equal, the infor- with a wide variety of communication protocols mation security manager would like and services. This article describes the popular to be able to ignore how information protocols and services, discusses about their in- moves from one place to another. He tended uses and applications, and describes their security properties and characteristics. or she would like to be able to as- Therefore, this should make life easier for the in- sume that information can be put formation security person faced with the need to into a pipe at point A and have it understand which way to go to best protect criti- cal/sensitive information.

come out reliably only at B. Of course, in the real world of the modern integrated network, this is not the case. In this world, the traffic is vulnerable to eavesdropping, misdirection, interference, contamination, alter- ation, and even total loss. On the other hand, relatively little of this happens; the vast majority of all information is delivered when and how it is intended and without any compromise. This happens, in part, despite the way that the information is moved and, in part, because of how it is moved. The various protocols and services have different security properties and qualities. Some provide error detection, corrective action such as retransmission, error correction, guaranteed delivery, and even information hiding. The different levels of service exist because they have different costs and performances. They exist because different traffic, applications, and environments have different requirements. For example, the transfer of a program file has a requirement for bit-for-bit integrity; in some cases, losing a bit is as bad as losing the entire file. On the other hand, a few seconds, or even tens of seconds, of delay in the transfer of the file may have little impact. However, if one is moving voice traffic, the loss of tens of bits may be perfectly acceptable, while a delay in seconds is intolera- ble. These costs must be balanced against the requirements of the application and the environment. While the balance between performance and cost is often struck without regard to security, the reality is that there are security differences. The balance between performance, cost, and security is the province of the information security manager. Therefore, it is necessary to understand the properties and characteristics of the protocols so that he or she can make the necessary trade-offs or evaluate those that have already been made. Finally, all protocols have limitations and many have fundamental vulnerabilities. Implementations of protocols can compensate for such vulnerabilities only in part. Implementers may be faced with difficult design choices and they may make errors resulting in implementation-induced vulnerabilities. The manager must understand these so that he or she will know when and how to compensate.

PROTOCOLS A protocol is an agreed-upon set of rules or conventions for communi- cating between two or more parties. “Hello” and “goodbye” for begin- ning and ending voice phone calls is an example of a simple protocol. A slightly more sophisticated protocol might include lines that begin with tags like “This is (name) calling.” Protocols are to codes as sentences and paragraphs are to words. In a protocol, the parties may agree to addressing, codes, format, packet size, speed, message order, error detection and correction, acknowledgments, key exchange, and other things.

This article deals with a number of common protocols. It will describe their intended use or application, characteristics, design choices, and limitations.

Internet Protocol The Internet Protocol, IP (pronounced “eye pea”), is a primitive and application-independent protocol for addressing and routing packets of data within a network. It is the IP in TCP/IP, the protocol suite, that is used in and defines the Internet. It is intended for use in a relatively flat, mesh, broadcast, connectionless, packet-switched nets like the Internet. IP is analogous to a postcard in the 18th century. The sender wrote the message on one side of the card, the address and return address on the other, and then gave it to someone who was going in the general direc- tion of the intended recipient. The message was not confidential; every- one who handled it could read it and might even make an undetected change to it. IP is a “best-efforts” protocol; it does not guarantee message delivery, nor does it provide any evidence as to whether or not the message was delivered. It is unchecked; the receiver does not know whether or not he received the entire intended message or whether or not it is correct. The addresses are unreliable; the sender cannot be sure that the message will go only where he intends or even where he intends. The receiver cannot be sure that the message came from the address specified as the return address in the packet. The protocol does not provide any checking or hiding. If the application requires these, they must be implied or specified someplace else, usually in a higher (i.e., closer to the application) protocol layer. IP specifies the addresses of the sending or receiving hardware device,1 but if that device supports multiple applications, IP does not specify for which of those it is intended. The IP protocol uses 32 bit addresses. However, the use or meaning of the bits within the address depends on the size and use of the network. Addresses are divided into five classes. Each class represents a different design choice between the number of networks and the number of addressable devices within the class. Class A addresses are used for very large networks where the number of such networks is expected to be low but the number of addressable devices is expected to be very high. Class A addresses are used for nation states and other very large domains such as .mil, .gov, and .com. A zero in bit position 0 of an address specifies it as a class A address. Positions 1 through 7 are used to specify the network, and positions 8 through 31 are used to specify devices within the network. Class C is used for networks where the possible number of networks is expected to be high but the number of addressable devices in each net is less than 128. Thus, in general, class B

EXHIBIT 1 — IP Network Address Formats

Device Network Class Description Address Class Network Address Address

A National 0 in bit 0 1–7 8–31 B Enterprise 10 in bits 0–1 2–15 16–31 C LAN 110 in 0–2 3–23 24–31 D Multicast 1110 in 0–3 4–31 E Reserved 1111 in 0–3 is used for enterprises, states, provinces, or municipalities, and class C is used for LANs. Class D is used for multicasting, and class E is reserved for future uses (see Exhibit 1). One often sees IP addresses written as nnn.nnn.nnn.nnn. While security is certainly not IP’s long suit, it is responsible for much of the success of the Internet. It is fast and simple. In practice, the security limitations of IP simply do not matter much. Applications rely on higher-level protocols for security.

Internet Protocol v6.0 (IPng) IPv6, or next-generation IP is a backwardly compatible new version of IP. It is intended to permit the Internet to grow, both in terms of the number of addressable devices, particularly class A addresses, and in quantity of traffic. It expands the address to 128 bits, simplifies the format header, improves the support for extensions and options, adds a quality-of-service capability, and adds address authentication and message confidenti- ality and integrity. IPv6 also formalizes the concepts of packet, node, router, host, link, and neighbors that were only loosely defined in IPv4. In other words, IPng addresses most of the limitations of IP, specifically including the security limitations. It provides for the use of encryption to ensure that information goes only where it is intended to go. This is called secure-IP (IPsec). Secure-IP can be used for point-to-point security across an arbitrary network. More often, it is used to carve virtual private networks (VPNs) or secure virtual networks (SVNs) out of such arbitrary networks.2 Many of the implementations of secure-IP are still proprietary and do not guarantee interoperability with all other such implementations.

User Datagram Protocol (UDP) UDP is similar to IP in that it is connectionless, offers “best-effort” delivery service, and is similar to TCP in that it is both checked and application speciﬁc. Exhibit 2 shows the format of the UDP datagram. Unless the UDP source port is on the same device as the destination port, the UDP packet

EXHIBIT 2 — UDP Datagram

Bit Positions Usage

0–15 Source port address 16–31 Destination port address 32–47 Message length (n) 48–63 Checksum 64–n Data will be encapsulated in an IP packet. The IP address will specify the physical device while the UDP address will specify the logical port or application on the device. UDP implements the abstraction of “port,” a named logical connection or interface to a specific application or service within a device. Ports are identified by a positive integer. Port identity is local to a device (i.e., the use or meaning of port number is not global). A given port number can refer to any application that the sender and receiver agreed on. However, by convention and repeated use, certain port numbers have become identified with certain applications. Exhibit 3 has examples of some of these conventional port assignments.

Transmission Control Protocol (TCP) TCP is a sophisticated composition of IP that compensates for many of its limitations. It is a connection-oriented protocol that enables two applications to exchange streams of data synchronously and simultaneously in both directions. It guarantees both the delivery and order of the packets. Since packets are given a sequence number, missing packets will be detected and packets can be delivered in the same order in which they were sent; lost packets can be automatically resent. TCP also adapts to the la- tency of the network. It uses control ﬂags to enable the receiver to automatically slow the sender so as not to overﬂow the buffers of the receiver.

EXHIBIT 3 — Sample UDP Ports

Port Number Application Description

23 Telnet 53 DNS Domain name service 43 Whois 69 TFTP Trivial ﬁle transfer service 80 HTTP Web service 119 Net News 137 Netbios name service 138 Netbios datagrams 139 Netbios session data

TCP does not make the origin address reliable. The sequence number feature of TCP resists address spoofing; however, it does not make it im- possible. Instances of attackers pretending to be trusted nodes have been reported to have toolkits that encapsulate the necessary work and special knowledge to implement such attacks. Like many packet-switched protocols, TCP uses path diversity. This means that some of the meaning of the traffic may not be available to an eavesdropper. However, eavesdropping is still possible. For example, user identifiers and passphrases usually move in the same packet. “Pass- word grabber” programs have been detected in the network. These programs simply store the first 256 or 512 bits of packets on the assumption that many will contain passwords. Finally, like most statefull protocols, some TCP implementations are vulnerable to denial-of-service attacks. One such attack is called SYN flooding. Requests for sessions, SYN flags, are sent to the target, but the acknowledgments are ignored. The target allocates memory to these requests and is overwhelmed.

Telnet Telnet describes how commands and data are passed from one machine on the network to another over a TCP/IP connection. It is described in RFC 855. It is used to make a terminal or printer on one machine and an operating system or application on another appear to be local to each other. The user invokes the telnet client by entering its name or clicking its icon on the local system and giving the name or address and port number of the system or application that the user wishes to use. The telnet client must listen to the keyboard and send the characters entered by the user across the TCP connection to the server. It listens to the TCP connection and displays the trafﬁc on the user’s terminal screen. The client and server use an escape sequence to distinguish between user data and their communication with each other. The telnet service is a frequent target of attack. By default, the telnet service listens for log-in requests on port 23. Connecting this port to the public network can make the system and the network vulnerable to attack. When connected to the public net, this port should expect strong authentication or accept only encrypted trafﬁc.

File Transfer Protocol (FTP) FTP is the protocol used on the Internet for transferring files between two systems. It divides a file into IP packets for sending it across the In- ternet. The object of the transfer is a file. The protocol provides automatic checking and retransmission to provide for bit-for-bit integrity. (See File Transfer Service below.)

Serial Line Internet Protocol (SLIP) SLIP is a protocol for sending IP packets over a serial line connection. It is described in RFC 1055. SLIP is often used to extend the path from an IP-addressable device (like a router at an ISP) across a serial connection (e.g., a dial connection), to a non-IP device (e.g., a serial port on a PC). It is a mechanism for attaching non-IP devices to an IP network. SLIP encapsulates the IP packet and bits in the code used on the serial line. In the process, the packet may gain some redundancy and error correction. However, the protocol itself does not provide any error detection or correction. This means that errors may not be detected until the trafﬁc gets to a higher layer. Because SLIP is usually used over relatively slow (56 KB) lines, this may make error correction at that layer expensive. On the other hand, the signaling over modern modems is fairly robust. Sim- ilarly, SLIP trafﬁc may gain some compression from devices (e.g., modems) in the path, but it does not provide any compression of its own. Because the serial line has only two endpoints, the protocol does not contain any address information; that is, the addresses are implicit. How- ever, this limits the connection to one application, and any distinctions in the intended use of the line must be handled at a higher layer. Since SLIP is used on point-to-point connections, it may be slightly less vulnerable to eavesdropping than a shared media connection like Ether- net. However, because it is closer to the endpoint, the data may be more meaningful. This observation also applies to PPP, as discussed below.

Point-to-Point Protocol (PPP) PPP is used for applications and environments similar to those for SLIP, but is more sophisticated. It is described in RFC 1661, July 1994. It is the Internet standard for transmission of IP packets over serial lines. It is more robust than SLIP and provides error detection features. It supports both asynchronous and synchronous lines. It is intended for simple links that deliver packets between two peers. It enables the transmission of multiple network-layer protocols (e.g., ip, ipx, spx) simultaneously over a single link. For example, a PC might run a browser, a Notes client, and an e-mail client over a single link to the network. To facilitate all this, PPP has a Link Control Protocol (LCP) to negotiate encapsulation formats, format options, and limits on packet format. Optionally, a PPP node can require that its partner authenticate itself using CHAP or PAP. This authentication takes place after the link is set up and before any trafﬁc can ﬂow. (See CHAP and PAP below.)

Hypertext Transfer Protocol (HTTP) HTTP is used to move data objects, called pages, between client applications, called browsers, running on one machine, and server applications,

usually on another. HTTP is the protocol that is used on and that defines the World Wide Web. The pages moved by HTTP are compound data objects composed of other data and objects. Pages are specified in a language called hypertext markup language, or html. Html specifies the appearance of the page and provides for pages to be associated with one another by cross-references called hyperlinks. The fundamental assumption of HTTP is that the pages are public and that no data hiding or address reliability is necessary. However, because many electronic commerce applications are done on the World Wide Web, other protocols, described below, have been defined and implemented.

SECURITY PROTOCOLS Most of the traffic that moves in the primitive TCP/IP protocol suite is public; that is, none of the value of the data derives from the confidenti- ality of the data. Therefore, the fact that the protocols do not provide any data hiding does not hurt anything. The protocols do not add any security, but the data does not need it. However, there is some traffic that is sensitive to disclosure and requires more security than the primitive protocols provide. The absolute amount of this traffic is clearly growing and its proportion may be growing. In most cases, the necessary hiding of this data is done in alternate or higher-level protocols. A number of these secure protocols have been defined and are rapidly being implemented and deployed. This section describes some of those protocols.

Secure Socket Layer (SSL) Arguably, the most widely used secure protocol is SSL. It is intended for use in client/server applications in general. More specifically, it is widely used between browsers and Web servers on the WWW. It uses a hybrid of symmetric and asymmetric key cryptography in which a symmetric algorithm is used to hide the traffic, and an asymmetric algorithm (RSA) is used to negotiate the symmetric keys. SSL is a session-oriented protocol; that is, it is used to establish a secure connection between the client and the server that lasts for the life of the session or until terminated by the application. SSL comes in two flavors and a number of variations. At the moment, the most widely used of the two flavors is one-way SSL. In this implementation, the server side has a private key, a corresponding public key, and a certificate for that key-pair. The server offers its public key to the client. After reconciling the certificate to satisfy itself as to the identity of the server, the client uses the public key to securely negotiate a session key with the server. Once the session key is in use, both the client and the server can be confident that only the other can see the traffic.

The client side has a public key for the key-pair that was used to sign the certificate and can use this key to verify the bind between the key- pair and the identity of the server. Thus, the one-way protocol provides for the authentication of the server to the client, but not the other way around. If the server cares about the identity of the client, it must use the secure session to collect evidence about the identity of the client. This evidence is normally in the form of a user identifier and a pass phrase or similar, previously shared, secret. The other flavor of SSL is two-way SSL. In this implementation, both the client and the server know the public key of the other and have a certificate for this key. In most instances, the client’s certificate is issued by the server, while the server’s certificate is issued by a mutually trusted third party.

Secure-HTTP S-HTTP is a secure version of HTTP designed to move individual pages securely on the World Wide Web. It is page oriented as opposed to SSL, which is connection or session oriented. Most browsers (thin clients) that implement SSL also implement S-HTTP, may share key-management code, and may be used in ways that are not readily distinguishable to the end user. In other applications, S-HTTP gets the nod where very high performance is required and where there is limited need to save state between the client and the server.

Secure File Transfer Protocol (S-FTP) Most of the applications of the primitive ﬁle transfer protocol are used to transfer public ﬁles in private networks. Much of it is characterized as “anonymous”; that is, one end of the connection may not even recognize the other. However, as the net spreads, FTP is increasingly used to move private data in public networks. S-FTP adds encryption to the FTP protocol to add data hiding to the integrity checking provided in the base protocol.

Secure Electronic Transaction (SET) SET is a special protocol developed by the credit card companies and vendors and intended for use in multi-party ﬁnancial transactions like credit card transactions across the Internet. Not only does it provide for hiding credit card numbers as they cross the network, but also for hiding them from some of the parties to the transaction and for protecting against replay. One of the limitations of SSL when used for credit card numbers is that the merchant must become party to the entire credit card number and must make a record of it to use in case of later disputes. This creates a

vulnerability to the disclosure and reuse of the credit card number. SET uses public key cryptography to guarantee that the merchant will be paid without having to know or protect the credit card number.

Point-to-Point Tunneling Protocol (PPTP) PPTP is a protocol (from the PPTP Forum) for hiding the information in IP packets, including the addresses.3 It is used to connect (portable computer) clients across the dial-switched point-to-point network to the In- ternet and then to a (MS) gateway server to a private (enterprise) network or to (MS) servers on such a network. As its name implies, it is a point-to-point protocol. It is useful for implementing end-to-end secure virtual networks (SVNs), but less so for implementing any-gateway-to- any-gateway virtual private networks (VPNs). It includes the ability to:

• query the status of comm servers • provide in-band management • allocate channels and place outgoing calls • notify server on incoming calls • transmit and receive user data with ﬂow control in both directions • notify server on disconnected calls

One major advantage of PPTP is that is included in MS 32-bit operating systems. (At this writing, the client-side software is included on 32-bit MS Windows operating systems Dial Up Networking [rel. 1.2 and 1.3]. The server-side software is included in the NT Server operating system. See L2TP below.) A limitation of PPTP, when compared to Secure-IP or SSL, is that it does not provide authentication of the endpoints. That is, the nodes know that other nodes cannot see the data passing between, but must use other mechanisms to authenticate addresses or user identities.

Layer 2 Forwarding (L2F) L2F is another mechanism for hiding information on the Internet. The encryption is provided from the point where the dial-switched point-to- point network connects to the Internet service provider (ISP) to the gateway on the private network. The advantage is that no additional software is required on the client computer; the disadvantage is that the data is protected only on the Internet and not on the dial-switched network. L2F is a router-to-router protocol used to protect data from acquisition by an ISP provider, across the public digital packet-switched network (Inter- net), to receipt by a private network. It is used by the ISP to provide data- hiding servers to its clients. Because the protocol is implemented in the routers (Cisco), its details and management are hidden from the end users.

Layer 2 Tunneling Protocol (L2TP) L2TP is a proposal by Microsoft and Cisco to provide a client-to-gateway data-hiding facility that can be operated by the ISP. It responds to the limitations of PPTP (must be operated by the owner of the gateway) and L2F (does not protect data on the dial-switched point-to-point net). Such a solution could protect the data on both parts of the public network, but as a service provided by the ISP rather than by the operator of the private network.

Secure Internet Protocol (Secure-IP or IPsec) IPsec is a set of protocols that provides for end-to-end encryption of the IP packets. It is being developed by the Internet Engineering Task Force (IETF). It is to be used to bind endpoints to one another and to implement VPNs and SVNs.

Internet Security Association Key Management Protocol (ISAKMP) ISAKMP is a proposal for a public-key, certiﬁcate-based key management protocol for use with IPsec. In order to establish a secure session, the user must have both a certiﬁcate and the corresponding key; and because the session will not be vulnerable to replay or eavesdropping, ISAKMP provides strong authentication. Moreover, because the same mechanism can be used for encryption as well as for authentication, it provides economy of administration.

Password Authentication Protocol (PAP) As noted above, PPP provides for the parties to identify and authenticate each other. One of the protocols for doing this is PAP. (See also CHAP below.) PAP works very much like traditional log-in using a shared secret. A sends a prompt or a request for authentication to B, and B responds with an identiﬁer and a shared secret. If the pair of values meets A’s expectation, then A acknowledges B. This protocol is vulnerable to a replay attack. It is also vulnerable to abuse of B’s identity by a privileged user of A.

Challenge Handshake Authentication Protocol (CHAP) CHAP is a standard challenge-response, peer-to-peer authentication mechanism. System A chooses a random number and passes it to B. B encrypts this challenge under a secret shared with A and returns it to A. A also computes the value of the challenge encrypted under the shared secret and compares this value to the value returned by B. If this response meets A’s expectation, then A acknowledges B.

Many implementations of PPP/CHAP provide that the remote party be periodically re-authenticated by sending a new challenge. This resists any attempt at “session stealing.”

SERVICES Telnet File Transfer. FTP is the name of a protocol, but it is also the name of a service that uses the protocol to deliver files. The service is symmetric in that either the server or the client can initiate a transfer in either direc- tion, either can get a file or send a file, and either can do a get or a put. The client may itself be a server. The server may or may not recognize its user, and may or may not restrict access to the available files. Where the server does restrict access to the available files, it usually does so through the use of the control facilities of the underlying file system. If the file server is built on the UNIX operating system and file system or the Windows operating systems, then it will use the rules-based file access controls of the file system. If the server is built on the NT operating system, then it will use the object-oriented controls of the NT file system. If the file service is built on MVS — and yes, that does happen — then it is the optional access control facility of MVS that will be used.

Secure Shell (SSH 2) Secure Shell is a UNIX-to-UNIX client/server program that uses strong cryptography for protecting all transmitted data, including passwords, bi- nary files, and administrative commands, between systems on a network. One can think of it as a client/server command processor or shell. While it is used primarily for system management, it should not be limited to this application. SSH 2 implements Secure-IP and ISAKMP at the application layer, as opposed to the network layer, to provide a secure network computing environment. It provides node identification and authentication, node-to- node encryption, and secure command and file transfer. It compensates for most of the protocol limitations noted above. It is now preferred to and used in place of more limited or application-specific protocols or implementations such as S-FTP.

CONCLUSIONS Courtney’s ﬁrst law says that nothing useful can be said about the security of a mechanism except in the context of an application and an environment. Of course, the converse of that law says that in such a context, one can say quite a great deal. The Internet is an open — not to say hostile — environment in which almost everything is permitted. It is deﬁned almost exclusively by its ad-

dresses and addressing schema and by the protocols that are honored in it. Little else is reliable. Nonetheless, most sensitive applications can be done there as long as one understands the properties and limitations of those protocols and carefully chooses among them. It has been noted that there are a large number of protocols defined and implemented on the Internet. No small number of them are fully adequate for all applications. On the other hand, the loss in performance, flexibility, generality, and function in order to use those that are secure for the intended application and environment are small. What is more, as the cost of performance falls, the differences become even less significant. The information security manager must understand the needs of his or her applications, and know the tools, protocols, and what is possible in terms of security. Then those protocols and implementations must be chosen and applied carefully.

Notes 1. There is a convention of referring to all network addressable devices as “hosts.” Such usage in other doc- uments equates to the use of device or addressable device here. IPv6 deﬁnes “host.” 2. VPN is used here to refer to the use of encryption to connect private networks across the public network, gateway-to-gateway. SVN is used to refer to the use of encryption to talk securely, end-to-end, across arbitrary networks. While the term “VPN” is sometimes used to describe both applications, different implementations of Secure-IP may be required for the two applications. 3. Tunneling is a form of encapsulation in which the encrypted package, the passenger, is encapsulated inside a datagram of the carrier protocol.

William Hugh Murray is an executive consultant for Deloitte and Touche LLP in New Canaan, CT.