URL Categories PacketShaper 11.6 Third Party Copyright Notices © 2016 Blue Coat Systems, Inc. All rights reserved. BLUE COAT, PROXYSG, PACKETSHAPER, CACHEFLOW, INTELLIGENCECENTER, CACHEOS, CACHEPULSE, CROSSBEAM, K9, DRTR, MACH5, PACKETWISE, POLICYCENTER, PROXYAV, PROXYCLIENT, SGOS, WEBPULSE, SOLERA NETWORKS, DEEPSEE, DS APPLIANCE, SEE EVERYTHING. KNOW EVERYTHING., SECURITY EMPOWERS BUSINESS, BLUETOUCH, the Blue Coat shield, K9, and Solera Networks logos and other Blue Coat logos are registered trademarks or trademarks of Blue Coat Systems, Inc. or its affiliates in the U.S. and certain other countries. This list may not be complete, and the absence of a trademark from this list does not mean it is not a trademark of Blue Coat or that Blue Coat has stopped using the trade- mark. All other trademarks mentioned in this document owned by third parties are the property of their respective owners. This document is for informational purposes only. BLUE COAT MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT. BLUE COAT PRODUCTS, TECHNICAL SERVICES, AND ANY OTHER TECHNICAL DATA REFERENCED IN THIS DOCUMENT ARE SUBJECT TO U.S. EXPORT CONTROL AND SANCTIONS LAWS, REGULATIONS AND REQUIREMENTS, AND MAY BE SUBJECT TO EXPORT OR IMPORT REGULATIONS IN OTHER COUNTRIES. YOU AGREE TO COMPLY STRICTLY WITH THESE LAWS, REGULATIONS AND REQUIREMENTS, AND ACKNOWLEDGE THAT YOU HAVE THE RESPONSIBILITY TO OBTAIN ANY LICENSES, PERMITS OR OTHER APPROVALS THAT MAY BE REQUIRED IN ORDER TO EXPORT, RE-EXPORT, TRANSFER IN COUNTRY OR IMPORT AFTER DELIVERY TO YOU. Americas: Blue Coat Systems, Inc. 384 Santa Trinita Avenue Sunnyvale, CA 94085 Rest of the World: Blue Coat Systems International SARL 3a Route des Arsenaux 1700 Fribourg, Switzerland 6/28/2016 URL Categories URL Categories Overview Web URLs can be grouped into various categories, such as social networking, gambling, pornography, news media, and shopping. PacketShaper is able to analyze URLs that users are requesting, determine what category the website belongs to, and classify the traffic into the appropriate category class. This gives you granular visibility into the type of web traffic on your network. You can also use PacketShaper’s control capabilities to assign policies based on the category. For example, you can assign a never-admit policy to all the category classes with adult content. For a complete list of the URL categories PacketShaper can identify, see URL Categories. The following steps and illustration describe PacketShaper's URL categorization and classification process. This process assumes the WebPulse feature is enabled and the traffic tree has been configured for category classes (see Category Clas- sification below). 1. When a user requests a web page, PacketShaper first looks in its in-memory URL cache to see if it has already categorized the URL. If it finds the URL in the cache, it jumps to step 4. If not, it proceeds to step 2. 2. PacketShaper sends the URL to WebPulse, a cloud-based service provided by Blue Coat that (a) receives URL categorization requests and (b) responds with the ID* of the category associated with the URL. 3. This URL and its category ID are then stored in the URL cache in the PacketShaper system memory. 4. PacketShaper looks in the WebPulse map file for the category name associated with the URL’s category ID. 5. PacketShaper classifies the traffic into the appropriate category class, and applies any policy that might be assigned to the class. * This example assumes the URL has a single category ID. However, WebPulse can assign up to four categories to a given URL. In addition to the URL category, WebPulse looks up the web application and operation. See WebPulse Overview for a description of the complete process. 3 URL Categories Feature WebPulse Express With WebPulse Express, the URL database is stored locally (available on the PS-S500 appliance only). The following steps and illustration describe the process for identifying URL categories when the URL database is on box. 1. When a user requests a web page, PacketShaper first looks in its on-box URL database for the URL. 2. If it finds the URL in the database, it jumps to step 4. If not, it proceeds to step 3. 3. PacketShaper looks in its in-memory URL caches to see if it has already identified the URL. If it finds the URL in the cache, it proceeds to step 4. If not, PacketShaper sends the URL to the WebPulse cloud to identify the URL category.The URL and its category ID is stored in the URL cache in the PacketShaper system memory. 4. PacketShaper looks in the WebPulse map files for the category name associated with the URL’s category ID. 5. PacketShaper classifies the traffic into the appropriate class, and applies any policy that might be assigned to the class. Category Classification Each HTTP request is categorized; this means a flow may jump across multiple classes during its lifetime. When a user browses a website, such as cnn.com, the browser attempts to resolve all the embedded links on that site to make them viewable; this could result in many class and category hits for a single website visit. SSL traffic is categorized based on the domain in the common name field of the SSL certificate. When URL categorization is enabled, you will have the ability to: l Create classes based on specific URL categories, and PacketShaper will then classify web traffic that corresponds to each of these categories into the appropriate class. l Auto-discover category classes. Note that you cannot auto-discover category classes at the top level of the traffic tree. 4 URL Categories A traffic tree can have a mix of category-based classes and web- application classes. Web-application classes are more specific than category-based classes and will naturally sort to the top of the traffic tree, as shown on the tree to the left. With a class tree that has both web- application and category-based classes, users can provide different treatment to a specific web application (such as Facebook) as opposed to other traffic in its URL category (Social Networking). But what if you prefer Facebook to be classified into the Social_Networking category class instead of the Facebook web application class? In situations where you don’t need special treatment for a web application, you will want your category classes to get the class hits, instead of the web application classes. You can do this by making the Categories class an exception, as shown on the tree to the right. The exclamation point (!) indicates the class was made an exception. The WebPulse service can assign up to four categories to a given URL. For instance, a URL and its corresponding flow could contain the following category IDs: catids[1 (Adult/Mature Content), 6(Nudity), 4(Sex Education)] When such a flow is sent through the class tree, a class hit will occur if any one of the four categories matches a category defined in a class. This capability enables matching of nested categories within the class tree. For instance, if Farmville is categorized as both Social Networking and Games it will hit the Games class in the following class tree: Inbound Social_Network Games Default 5 URL Categories Feature The above traffic tree would allow you to designate a different policy for social networking games than for other types of social networking traffic. WebPulse Interaction Blue Coat WebPulse contains a database of over 15 million entries, manages 85+ categories across 50 languages, and can return up to four categories per URL. URL categories are provided to the PacketShaper via requests to WebPulse ser- vice points located across the globe. Each service point is periodically pinged in order to ensure that categories are provided from the fastest service point. When a user requests a URL (such as a brand new website) that has not already been categorized by WebPulse, the WebPulse dynamic categorization service queries the target website and retrieves key pieces of the page's content and context. After analyzing elements of the web pages, including the language used, the dynamic real time rating (DRTR) ser- vice determines both a likely category and a confidence factor that the rating is correct. Note that the PacketShaper will not hold up traffic during this process. In the rare event that the DRTR service does not yield a result with a high con- fidence level, the category rating request for the particular page is labeled unknown. PacketShaper includes support for DRTR requests of plain HTTP traffic, but not HTTPS since only domain level inform- ation is available for SSL. URL Cache After WebPulse has looked up the category for a URL, PacketShaper stores the results in a URL cache in its system memory. Since serving from the cache is faster and more efficient than querying WebPulse, PacketShaper will always check the cache first before sending a query to WebPulse. WebPulse provides a “time to live” (TTL) for each cat- egorization response. The PacketShaper scales this TTL by a multiple of seven to improve its cache efficiency. For instance, if the WebPulse service returns a TTL of one day for a particular URL, the PacketShaper will cache that URL and its associated categories for seven days. The maximum cache size and number of entries in the cache varies by PacketShaper model. See Configuration Limits in PacketGuide. The URL cache is stored in system memory, so the contents will be removed if the PacketShaper is reset or loses power. To preserve the cache contents, PacketShaper automatically backs up the cache to the 9.1026/urlcat/cache folder every 24 hours (replacing the previous backup). After a PacketShaper reboot, the most recent URL cache backup is auto- matically copied into system memory.