UMEÅ UNIVERSITY June 4, 2021 Department of Computer Science Report 5DV199 TLS Decryption in passive monitoring system with server private key Degree Project: Bachelor of Science in Computing Science Name Emil Käck E-mail [email protected] Supervisor Jerry Eriksson Employer Polystar Contents 1 Introduction 1 1.1 Background . .1 2 Related Work 2 3 TLS Fundamentals 3 3.1 TLS Handshake Protocol . .3 3.2 TLS Record Protocol . .4 3.3 TLS hello extensions . .4 3.4 Cipher suites . .5 3.5 Forward secrecy . .5 3.6 The PRF algorithm . .6 3.6.1 Key Block . .7 3.7 Record Payload In CBC Mode . .7 4 Implementation delimitations 8 5 Proposed solution 9 5.1 Obtain premaster secret . .9 5.2 Derive session keys . 10 5.3 Decryption of traffic . 10 6 Experimental result 11 7 Discussion 13 7.1 Conclusion and Recommendations . 13 7.2 Future Work . 14 7.3 Personal Reflections . 14 Abstract Many network operators need to be able to ensure that customers get the level of service they pay for. To avoid bandwidth and server performance bot- tlenecks, and easily troubleshoot network problems, the network providers need to be able to see what payload data is sent. Modern networks encrypt data when sending it between nodes that makes passive monitoring more complex. A prevalent encryption mechanism on an IP-based network is TLS that needs to be decrypted. This article’s purpose is to check if it is possible to decrypt TLS traffic in a passive monitoring system with the server’s private key. This is done by implementing a decryptor in a passive monitoring system in the programming language Java. The implemented solution intercepts the traffic, takes out relevant data from the traffic, and derives the session key from that data. How this is done is dependent on what cipher suite is used for the session. Because of delimitations and lack of time the solution is only able to decrypt the cipher suite TLS_RSA_WITH_AES_128_CBC_SHA256. The result showed that it is possible to decrypt TLS traffic and should be possible for more than the specified cipher suite. But there exists a major problem that’s called forward secrecy. This is used in the key exchange algorithm called Diffie–Hellman and makes it impossible to decrypt with only server private key. The conclusion is that it is possible but because of forward secrecy, it is not recommended. TLS 1.3 only uses cipher suites with the key exchange algorithm Diffie–Hellman and the forward secrecy functionality is important for security. Acknowledgement I want to thank my friends William, Magnus, and Tobias for all the fun times over these 3 years of studying. I also want to thank them for our nice GeoGuesser breaks under the course. It was fun breathers between all coding and writing. I also want to thank Jerry for the help with this article. TLS Decryption Introduction 1 Introduction Passive network monitoring is an essential tool to ensure quality and reliabil- ity for telecommunication networks. Passive monitoring is a technique used to capture traffic from a network by copying traffic. By monitoring traffic between network nodes, the network operator can ensure that its customers get the level of service they pay for. The network providers can also avoid bandwidth and server performance bottlenecks, easily troubleshoot network problems, and more to ensure quality for the users. Traditionally, the signal- ing between core network nodes has been non-encrypted and the separation of the telecommunication domain from the public internet has been deemed sufficient from a security and privacy point of view. This is no longer the case, modern networks encrypt data when sending it between nodes which makes passive monitoring much more complex. Because of this, many net- work providers are not able to monitor the network where the encryption applies. Transport Layer Security (TLS) is a cryptographic protocol de- signed to provide communications security over a computer network and is a prevalent encryption mechanism on IP-based network interfaces that need to be decrypted. This thesis work aims to mitigate the problem of encrypted signaling on the above-mentioned interfaces by providing an approach on how decryption of signaling can be done inside a passive monitoring system. This helps network operators ensure the quality of service to the customers. This is a relevant problem to Polystar that is a company selling products to network providers for monitoring traffic. The TLS traffic prevalent for Polystar is SIP-trunking that uses TLS for encryption. A more detailed explanation about this is in the Section 1.1. The suggested method is tested to verify that it works and check if there exists any problem with the presented solution. The research question for this thesis is, • can TLS traffic effectively be decrypted in a passive monitoring system with only the server’s private key? A program is implemented in Java for the decryption of TLS data. The program is tested towards test data and should be able to decrypt all the data with the server’s private key. The implemented program is integrated into Polystars system to see if it can work in real use. 1.1 Background One of the IP-based networks is Voice Over IP(VoIP)[1] that is used for the delivery of voice communications and multimedia sessions over internet protocol networks. Before VoIP, voice has been transmitted over the public switched telephone network (PSTN). The investment over IP-based networks (public and private) produced the interest in transmitting voice over IP. VoIP converts the user’s voice to a digital signal. SIP trunking [2] is a VoIP technology based on Session Initiation Proto- col (SIP)[3] that delivers telephone services and unified communications to Emil Käck 1 June 4, 2021 TLS Decryption Related Work customers. SIP trunking provides network access to multiple clients simul- taneously. SIP is an application-layer control protocol that can establish, modify, and terminate multimedia sessions that include voice, video, and messaging applications [3]. It can also invite participants to already existing sessions, e.g. multicast conferences. Media can be added to an existing session. It incorporates many elements of the Hypertext Transfer Protocol(HTTP) and Simple Mail Transfer Protocol(SMTP). SIP works in conjunction with sev- eral other protocols that specify and carry the session media. It is often used with Real-time Transport Protocol (RTP) for transporting real-time data, Real-Time streaming protocol (RTSP) for controlling delivery of streaming media, Media Gateway Control Protocol (MEGACO) for controlling gate- way to the Public Switched Telephone Network (PSTN), and the Session Description Protocol (SDP) for describing multimedia session. SIP uses a resource called SIP or SIPS URI(Uniform Resource Identi- fier)[4] that provide a simple and extensible mean for identifying a com- munication resource. A SIPS URI specifies that the resource is contacted securely. This means that TLS is to be used between user agent client (UAC) and the domain that owns the URI, this means that in the monitoring sys- tem the data received is encrypted with a TLS Cipher suite[5] making it impossible to monitor without access to the private key. 2 Related Work The article Passive, Transparent, and Selective TLS Decryption for Network Security Monitoring [6] introduces a new solution for monitoring the network to ensure quality. The solution is based on studies which show that passive man-in-the-middle (MitM) proxies for monitoring reduces connection secu- rity and potentially introduces additional attack vectors to the network. It also lets users retain privacy for chosen connections. The solution is that the cooperative approach in which end-hosts as cryptographic end-points provide TLS key material to NMS (network monitoring system) for decryp- tion. The problem before is that the encrypted data prevented the NMS from analyzing the data. By giving the NMS the needed key material it can decrypt the data and handle it. The result the paper present is that the runtime increased by a factor of 2.5 compared to analyzing cleartexts only. The complexity analysis shows that it needs less computational resources than MitM proxies. The article also concluded that it could decrypt 99.9% of all observed TLS connections if it has a 40ms delay for the data. The important part is by making the end-host receiving all key-material to filter it and send it to the NMS, it can decrypt almost all TLS data and decrease the security risks. This is related to this article thesis because it solves the problem but differently, in the article, it is more implemented in the endpoint while this thesis focuses on only analyzing the traffic with no help from the endpoints. In many cases, the companies that own the server don’t want other companies implementing programs in their server. Emil Käck 2 June 4, 2021 TLS Decryption TLS Fundamentals 3 TLS Fundamentals TLS (Transport layer security)[5] is used to provide privacy and data in- tegrity between two communicating applications. It is divided into two lay- ers, the TLS Record Protocol and the TLS Handshake Protocol. The TLS Record Protocol provides connection security that has two properties. The first is that the connection is private. It uses symmetric cryptography for data encryption. The second is that the connection is reliable, message transport includes a message integrity check (MAC) to control that it is the correct application sending the data. The TLS Handshake Protocol allows the server and client to authenticate each other and negotiate an encryption algorithm and cryptographic keys before the application protocol transmits or receives its first byte of data. 3.1 TLS Handshake Protocol Before the data can be sent over TLS, the client and server exchange all the information required by both sides [5].