2 HOW THE INTERNET WORKS To become an expert on web security, you need a firm grasp of the internet’s under- lying web technologies and protocols. This chapter examines the Internet Protocol Suite, which dictates how computers exchange data over the web. You’ll also learn about stateful connections and encryption, which are key elements of the modern web. I’ll highlight where security holes tend to appear along the way. The Internet Protocol Suite In the early days of the internet, data exchange wasn’t reliable. The first message sent over the Advanced Research Projects Agency Network (ARPANET), the predecessor to the internet, was a LOGIN command destined for a remote computer at Stanford University. The network sent the first two letters, LO, and then crashed. This was a problem for the US military, which was looking for a way to connect remote computers so that they could continue to exchange information even if a Soviet nuclear strike took various parts of the network offline. To address this problem, the network engineers developed the Transmission Control Protocol (TCP) to ensure a reliable exchange of information between computers. TCP is one of about 20 network protocols that are collectively referred to as the internet protocol suite. When a computer sends a message to another machine via TCP, the message is split into data packets that are sent toward their eventual destination with a destination address. The com- puters that make up the internet push each packet toward the destination without having to process the whole message. Once the recipient computer receives the packets, it assembles them back into a usable order according to the sequence number on each packet. Every time the recipient receives a packet, it sends a receipt. If the recipient fails to acknowledge receipt of a packet, the sender resends that packet, pos- sibly along a different network path. In this way, TCP allows computers to deliver data across a network that is expected to be unreliable. TCP has undergone significant improvements as the internet has grown. Packets are now sent with a checksum that allows recipients to detect data corruption and determine whether packets need to be resent. Senders also preemptively adjust the rate at which they send data according to how fast it’s being consumed. (Internet servers are usually magnitudes more power- ful than the clients that receive their messages, so they need to be careful not to overwhelm the client’s capacity.) NOTE TCP remains the most common protocol because of its delivery guarantees, but nowadays, several other protocols are also used over the internet. The User Datagram Protocol (UDP), for instance, is a newer protocol that deliberately allows packets to be dropped so that data can be streamed at a constant rate. UDP is commonly used for streaming live video, since consumers prefer a few dropped frames over having their feed delayed when the network gets congested. Internet Protocol Addresses Data packets on the internet are sent to Internet Protocol (IP) addresses, num- bers assigned to individual internet-connected computers. Each IP address must be unique, so new IP addresses are issued in a structured fashion. At the highest level, the Internet Corporation for Assigned Names and Numbers (ICANN) allots blocks of IP addresses to regional authorities. These regional authorities then grant the blocks of addresses to internet service pro- viders (ISPs) and hosting companies within their region. When you connect your browser to the internet, your ISP assigns your computer an IP address that stays fixed for a few months. (ISPs tend to rotate IP addresses for clients periodically.) Similarly, companies that host content on the internet are assigned an IP address for each server they connect to the network. IP addresses are binary numbers, generally written in IP version 4 (IPv4) syntax, which allows for 232 (4,294,967,296) addresses. Google’s 6 Chapter 2 domain name server, for instance, has the address 8.8.8.8. Because IPv4 addresses are getting used up at a rate that isn’t sustainable, the internet is shifting to IP version 6 (IPv6) addresses to allow for more connected devices, represented as eight groups of four hexadecimal digits separated by colons (for example: 2001:0db8:0000:0042:0000:8a2e:0370:7334). The Domain Name System Browsers and other internet-connected software can recognize and route traffic to IP addresses, but IP addresses aren’t particularly memorable for humans. To make website addresses friendlier to users, we use a global directory called the Domain Name System (DNS) to translate human-readable domains like example.com to IP addresses like 93.184.216.119. Domain names are simply placeholders for IP addresses. Domain names, like IP addresses, are unique, and have to be registered before use with private organizations called domain registrars. When browsers encounter a domain name for the first time, they use a local domain name server (typically hosted by an ISP) to look it up, and then cache the result to prevent time-consuming lookups in the future. This caching behavior means that new domains or changes to existing domains take a while to propagate on the internet. Exactly how long this propaga- tion takes is controlled by the time-to-live (TTL) variable, which is set on the DNS record and instructs DNS caches when to expire the record. DNS cach- ing enables a type of attack called DNS poisoning, whereby a local DNS cache is deliberately corrupted so that data is routed to a server controlled by an attacker. In addition to returning IP addresses for particular domains, domain name servers host records that can describe domain aliases via canonical name (CNAME) records that allow multiple domain names to point to the same IP address. DNS can also help route email by using mail exchange (MX) records. We’ll examine how DNS records can help combat unsolicited email (spam) in Chapter 16. Application Layer Protocols The technical standards used for communication on the internet, including TCP, are defined and developed by theInternet Engineering Task Force (IETF). These standards are called the internet protocol suite. TCP allows two comput- ers to reliably exchange data on the internet, but it doesn’t dictate how the data being sent should be interpreted. For that to happen, both computers need to agree to exchange information through another, higher-level pro- tocol in the suite. Protocols that build on top of TCP (or UDP) are called application layer protocols. Figure 2-1 illustrates how application layer proto- cols sit above TCP in the internet protocol suite. The lower-level protocols of the internet protocol suite provide basic data routing over a network, while the higher-level protocols in the applica- tion layer provide more structure for applications exchanging data. Many types of applications use TCP as a transport mechanism on the internet. How the Internet Works 7 For example, emails are sent using the Simple Mail Transport Protocol (SMTP), instant messaging software often uses the Extensible Messaging and Presence Protocol (XMPP), file servers make downloads available via the File Transfer Protocol (FTP), and web servers use the HyperText Transfer Protocol (HTTP). Because the web is our chief focus, let’s look at HTTP in more detail. Application layer DNS FTP HTTP IMAP POP SMTP SSH XMPP Transport layer TCP UDP Internet layer IPv4 IPv6 Network layer ARP MAC NDP OSPF PPP Figure 2-1: The various layers that make up the internet protocol suite HyperText Transfer Protocol Web servers use the HyperText Transfer Protocol (HTTP) to transport web pages and their resources to user agents such as web browsers. In an HTTP conversation, the user agent generates requests for particular resources. Web servers, expecting these requests, return responses containing either the requested resource, or an error code if the request can’t be fulfilled. Both HTTP requests and responses are plaintext messages, though they’re often sent in compressed and encrypted form. All of the exploits described in this book use HTTP in some fashion, so it’s worth knowing how the requests and responses that make up HTTP conversations work in detail. HTTP Requests An HTTP request sent by a browser consists of the following elements: Method Also known as a verb, this describes the action that the user agent wants the server to perform. Universal resource locator (URL) This describes the resource being manipulated or fetched. Headers These supply metadata such as the type of content the user agent is expecting or whether it accepts compressed responses. Body This optional component contains any extra data that needs to be sent to the server. Listing 2-1 shows an HTTP request. GETu http://example.com/v w User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36 x Accept: text/html,application/xhtml+xml,application/xml; */* 8 Chapter 2 Accept-Encoding: gzip, deflate Accept-Language: en-GB,en-US;q=0.9,en;q=0.8 Listing 2-1: A simple HTTP request The method u and the URL v appear on the first line. These are fol- lowed by HTTP headers on separate lines. The User-Agent header w tells the website the type of browser that is making the request. The Accept header x tells the website the type of content the browser is expecting. GET requests are the most common type of request on the internet— they request a particular resource on the web server, identified by a specific URL. The response to a GET request will contain a resource: perhaps a web page, an image, or even the results of a search request.