Apache-Trafficserver-Server Documentation

Apache-Trafﬁcserver-Server Documentation latest

7 21, 2017

Contents

1 1 1.1 Apache Trafﬁc Server?...... 1 1.2 Typographic Conventions...... 1 1.3 Other Resources...... 2

2 Getting Started 3 2.1 Introduction...... 3 2.2 Installation...... 4 2.3 Conﬁguration...... 6 2.4 Logging and Monitoring...... 9 2.5 Further Steps...... 9

3 Administrator’s Guide 11 3.1...... 11 3.2 Traffic Server...... 16 3.3...... 19 3.4 Interacting with Traffic Server...... 54 3.5 Security...... 54 3.6 Cache Storage...... 58 3.7 Plugins...... 64 3.8 Event and Error Monitoring...... 179 3.9 Configuring Traffic Server...... 213 3.10 Performance Tuning...... 214 3.11 Configuration Files...... 218 3.12 Audience...... 258

4 Developer’s Guide 259 4.1 Introduction...... 259 4.2 Release Process...... 260 4.3 Contributing to Traffic Server...... 262 4.4 Using Vagrant to Test Traffic Server...... 262 4.5 Debugging and Analysis...... 266 4.6 Cache Architecture...... 268 4.7 Plugin Development...... 291 4.8 Configuration Variable Implementation...... 376 4.9 API Reference...... 379 4.10 Continuous Integration...... 475

i 4.11 Documentation...... 475 4.12 Host Resolution Proposal...... 488

5 Appendices 491 5.1 Command Line Utilities...... 491 5.2 Frequently Asked Questions...... 505 5.3 Troubleshooting Tips...... 511 5.4 Glossary...... 514 5.5 HTTP Status Codes...... 516

6 Indices and tables 517

ii CHAPTER 1

Apache Trafﬁc Server?

Apache Traffic Server™ is a high-performance web proxy cache that improves network efficiency and performance by caching frequently-accessed information at the edge of the network. This brings content physically closer to end users, while enabling faster delivery and reduced bandwidth use. Traffic Server is designed to improve content delivery for enterprises, Internet service providers (ISPs), backbone providers, and large intranets by maximizing existing and available bandwidth. This manual will explore every aspect of installing, managing, extending, and troubleshooting Traffic Server.

Typographic Conventions

This documentation uses the following typographic conventions: Italic Used to introduce new terms on their initial appearance. Example: The Apache Traffic Server™ object storage is based on a cyclone buffer architecture. Cyclone buffers are a form of storage addressing in which a single writer continually reclaims the oldest allocations for use by new updates. Monospace Represents C/C++ language statements, commands, file paths, file content, and computer output. Example: The default installation prefix for Traffic Server is /usr/local/ts. Bracketed Monospace Represents variables for which you should substitute a value in file content or commands. Example: Running the command traffic_line -r will display the current value of a performance statistic, where is the statistic whose value you wish to view. Ellipsis Indicates the omission of irrelevant or unimportant information.

1 Apache-Trafﬁcserver-Server Documentation, latest

Other Resources

Websites

Official Website https://trafficserver.apache.org/ The official Apache Traffic Server™ project website is hosted by the Apache Software Foundation. Documen- tation, software downloads, community resource links, security announcements, and more are located, or linked to, at the site. Online Documentation https://docs.trafficserver.apache.org/ The most up to date version of the documentation is hosted at Read the Docs, with built-in search functionality. Documentation for past releases is also available. Jira Bug Tracker https://issues.apache.org/jira/browse/TS If you wish to report bugs, or look for open issues on which you may help contribute to the Traffic Server project, please visit the public Jira bug tracker site.

Mailing Lists

User List The user’s mailing list offers support and discussions oriented to users and administrators of the Trafﬁc Server software. Send an email to [email protected] to join the list. Developer List If you have questions about, or wish to discuss, the development of Trafﬁc Server, plugins for the server, or other developer-oriented matters, the developer list offers an active list of both core project members and external contributors. Send an email to [email protected] to join the list.

Internet Relay Chat (IRC)

The #traffic-server channel on irc.freenode.net is the ofﬁcial IRC resource for the Trafﬁc Server project, and boasts active discussions.

Community Forums

2 Chapter 1. CHAPTER 2

Getting Started

Introduction

Apache Traffic Server™ provides a high-performance and scalable software solution for both forward and reverse proxying of HTTP/HTTPS traffic, and may be configured to run in either or both modes simultaneously. This Getting Started guide explains the basic steps an administrator new to Traffic Server will need to perform to get the software up and running in a minimal configuration as quickly as possible.

Example Scenario

In this guide, we will use the fictional company Acme Widgets as a basis for the configuration examples. Acme Widgets has a product brochure website (assumed to reside at the domain www.acme.com) that performs very poorly. The content management software they chose takes an unbearable amount of time to generate pages on every request and their engineering team has chosen Traffic Server as a caching proxy layer to improve site performance. Separately, Acme Widgets has decided to use Traffic Server to help improve the performance of their office’s Internet access, which is hobbled by their reliance on an aging leased line and certain employees’ predilection for extracurric- ular web browsing.

Terminology

This guide uses some terms which may be unfamiliar to administrators new to proxy servers. Origin Server The server which generates the content you wish to proxy (and optionally cache) with Traffic Server. In a forward proxy configuration, the origin server may be any remote server to which a proxied client attempts to connect. In a reverse proxy configuration, the origin servers are typically a known set of servers for which you are using Traffic Server as a performance-enhancing caching layer. Reverse Proxy A reverse proxy appears to outside users as if it were the origin server, though it does not generate the content itself. Instead, it intercepts the requests and, based on the configured rules and contents of its cache, will either serve a cached copy of the requested content itself, or forward the request to the origin server, possibly caching the content returned for use with future requests.

3 Apache-Trafﬁcserver-Server Documentation, latest

Forward Proxy A forward proxy brokers access to external resources, intercepting all matching outbound traffic from a network. Forward proxies may be used to speed up external access for locations with slow connections (by caching the external resources and using those cached copies to service requests directly in the future), or may be used to restrict or monitor external access. Transparent Proxy A transparent proxy may be either a reverse or forward proxy (though nearly all reverse proxies are deployed transparently), the defining feature being the use of network routing to send requests through the proxy without any need for clients to configure themselves to do so, and often without the ability for those clients to bypass the proxy. For a more complete list of definitions, please see the Glossary.

Installation

As with many other software packages, you may have the option of installing Apache Traffic Server™ from your operating system distribution’s packages, or compiling and installing from the source code yourself. Distribution packages may lag behind the current stable release of Traffic Server, sometimes by a significant amount. While we will cover briefly the packages required for common operating system distributions, the remainder of this guide will be assuming the latest stable release of Traffic Server when discussing configuration parameters and available features.

Installing From Distribution Packages

It is recommended that you install from source code, as described in the section below, rather than rely on your distribution’s packages if you wish to have access to the latest features and ﬁxes in Trafﬁc Server.

Ubuntu

The Canonical repositories up through, and including, Utopic Unicorn only provide Trafﬁc Server v3.2.x. sudo apt-get install trafficserver

RHEL / CentOS

Traffic Server is available through the EPEL repositories. If you do not have those configured on your machine yet, you must install them first with the following: wget https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm sudo rpm-Uvh epel-release-7 *.rpm

Ensuring that you replace the release number with a value that is appropriate for your system. Once you have EPEL installed, you may install Trafﬁc Server itself. sudo yum install trafficserver

OmniOS (illumos)

OmniOS (an illumos-based distribution) provides the Trafﬁc Server package in its public OmniTI-MS publisher. sudo pkg set-publisher-g http://pkg.omniti.com/omniti-ms/ ms.omniti.com sudo pkg install omniti/server/trafficserver

4 Chapter 2. Getting Started Apache-Trafﬁcserver-Server Documentation, latest

The latest release published, at the time of this writing, is Trafﬁc Server v4.2.1.

Installing From Source Code

To install from source code, you will need to have the following tools and libraries on the machine used to build Traffic Server: • pkgconfig • libtool • gcc (>= 4.3 or clang > 3.0) • GNU make • openssl • tcl • expat • pcre • libcap • flex (for TPROXY) • hwloc • lua • curses (for traffic_top) • curl (for traffic_top) To build Traffic Server from a Git clone (the method we will be using in this guide), you will also need the following: • git • autoconf • automake In this guide, Traffic Server will be built to use the default nobody user and group and will be installed to /opt/ts. It is assumed that all of the dependencies are located in standard paths. If this is not the case, you may need to use the appropriate --with- options as detailed in the output of ./configure --help. 1. Clone the repository (you may skip this if you have downloaded an archive of the source code to build a specific official release of Traffic Server instead of the HEAD from source control):

git clone https://git-wip-us.apache.org/repos/asf/trafficserver.git

2. Change to the cloned (or unarchived) directory:

cd trafficserver/

3. If you have cloned the repository from Git, you will need to generate the configure script before proceeding:

autoreconf- if

4. Conﬁgure the source tree:

./configure--prefix=/opt/ts

2.2. Installation 5 Apache-Trafﬁcserver-Server Documentation, latest

5. Build Trafﬁc Server with the generated Makeﬁles, and test the results:

make make check

6. Install Trafﬁc Server to the conﬁgured location:

sudo make install

7. Optionally, you may ﬁnd it prudent to run the regression tests on your newly installed Trafﬁc Server:

cd/opt/ts sudo bin/traffic_server-R1

Conﬁguration

We will be tackling two separate configuration scenarios in the following sections. The first is the most common application of a performance-enhancing caching proxy for externally-facing websites, a transparent and caching reverse proxy which forwards all requests presented to it to a single origin address and caches the responses based on their cache control headers (as well as some simple heuristics for specific content types when cache control headers are not present). The second configuration we will review is a very basic transparent forward proxy, typically used in situations where you either need to improve the performance of a local network’s use of external resources, or you wish to have the capability of monitoring or filtering the traffic.

Conﬁguring A Reverse Proxy

A minimal reverse proxy configuration requires changes to only a few configuration files, which will all be located in the /opt/ts/etc/trafficserver directory if you have configured your installation per the instructions in Installing From Source Code above. For these examples, we will be assuming that Traffic Server is running on the same host machine as the origin website. This is not a requirement, and you may choose to run Traffic Server on an entirely different host, even connected to entirely different networks, as long as Traffic Server is able to reach the origin host.

Enable Reverse Proxying

Within the records.config configuration file, ensure that the following settings have been configured as shown below:

CONFIG proxy.config.http.cache.http INT1 CONFIG proxy.config.reverse_proxy.enabled INT1 CONFIG proxy.config.url_remap.remap_required INT1 CONFIG proxy.config.url_remap.pristine_host_hdr INT1 CONFIG proxy.config.http.server_ports STRING 8080

:ts:cv:‘proxy.config.http.cache.http‘ Enables caching of proxied HTTP requests. :ts:cv:‘proxy.config.reverse_proxy.enabled‘ Enables reverse proxying support. :ts:cv:‘proxy.config.url_remap.remap_required‘ This setting requires that a remap rule exist before Traffic Server will proxy the request and ensures that your proxy cannot be used to access the content of arbitrary websites (allowing someone of malicious intent to potentially mask their identity to an unknowning third party).

6 Chapter 2. Getting Started Apache-Trafﬁcserver-Server Documentation, latest

:ts:cv:‘proxy.config.url_remap.pristine_host_hdr‘ This setting causes Traffic Server to keep the Host: client request header intact which is necessary in cases where your origin servers may be performing domain-based virtual hosting, or taking other actions dependent upon the contents of that header. :ts:cv:‘proxy.config.http.server_ports‘ This configures Traffic Server to bind itself to the port 8080 for HTTP traffic.

Conﬁgure Origin Location

The previous settings enable reverse proxying (and prevent flagrant abuse of it), but now Traffic Server needs to know what to proxy. This is achieved by writing remap rules, which make use of the core Configuration Remap Plugin. For our Getting Started guide’s Acme Widgets example scenario, we have very simple needs and want little more than to proxy all requests to our single origin server. This is accomplished with the following rule added to the remap.config configuration:

regex_map http://(.*)/ http://localhost:80/

It is worth pausing at this point to note that in a reverse proxying scenario, it is Traffic Server itself which should be responding to HTTP requests made to your public domain. While you first configure and evaluate whether Traffic Server will meet your needs, your origin server will continue to reside at the public-facing domain name on the default ports, should you move your Traffic Server configuration into production your DNS records for the domain(s) you wish to proxy and cache should resolve to the host(s) running Traffic Server (in the event that you run it on a separate host from your origin). Your origin should be accessible at a different address (or bind to different ports if you are running both your origin service and Traffic Server on the same host) and should no longer receive requests sent to the primary domain on its default ports. In our Acme Widgets scenario, they ultimately decide to deploy Traffic Server at which point, since they are running both Traffic Server and their origin web server on the same host, they reconfigure their origin service to listen on port 8080 instead of the default, and change Traffic Server to bind to 80 itself. Updating the remap is thus required, and it should now be:

regex_map http://(.*)/ http://localhost:8080/

Now all requests made to www.acme.com are received by Trafﬁc Server which knows to proxy those requests to localhost:8080 if it cannot already serve them from its cache. Because we enabled pristine host headers earlier, the origin service will continue to receive Host: www.acme.com in the HTTP request.

Adjust Cache Parameters

The default Traffic Server configuration will provide a 256 MB disk cache, located in var/trafficserver/ underneath your install prefix. You may wish to adjust either or both of the size and location of this cache. This is done with the storage.config configuration file. In our example, Acme Widgets has dedicated a large storage pool on their cache server which is mounted at /cache. To use this, and to disable the default cache storage setting, the following will be the sole entry in storage.config:

/cache/trafficserver 500G

: Changes to the cache conﬁguration require a restart of Trafﬁc Server.

You may also wish to use raw devices, or partition your cache storage. Details on these features may be found in the section of the Administrator’s Guide.

2.3. Conﬁguration 7 Apache-Trafﬁcserver-Server Documentation, latest

Final Conﬁgurations

Once completed, the following conﬁguration ﬁles for Acme Widgets contain the following entries: records.config:

remap.config:

regex_map http://(.*)/ http://localhost:8080/

storage.config:

/cache/trafficserver 500G

Conﬁguring A Forward Proxy

Configuring a forward proxy with Traffic Server is somewhat more straightforward, though there are two main approaches for how to direct client traffic through the proxy: explicit or transparent. More detail on the process is also available in the Administrator’s Guide in the Forward Proxy section. This guide will cover only a minimal configuration.

Enable Forward Proxying

Contrary to a reverse proxy, where you have a deﬁned list of origin servers for which you wish to proxy (and optionally cache), a forward proxy is used to proxy (and optionally cache) for arbitrary remote hosts. As such, the following settings in records.config are the base conﬁguration for a minimal forward proxy:

CONFIG proxy.config.url_remap.remap_required INT0 CONFIG proxy.config.http.cache.http INT1

:ts:cv:‘proxy.config.url_remap.remap_required‘ Disables the requirement for a remap rule to exist and match the incoming request before Traffic Server will proxy the request to the remote host. :ts:cv:‘proxy.config.http.cache.http‘ Enables caching of proxied HTTP requests. If your installation will be strictly a forwarding proxy, then reverse proxying should be explicitly disabled:

CONFIG proxy.config.reverse_proxy.enabled INT0

Explicit Versus Transparent

With forward proxying enabled, the next step is deciding how clients will connect through the proxy server. Explicit forward proxying requires that every client application be configured (manually or through whatever configuration management system you may employ) to use the proxy. The presence of the proxy will be known to clients, and any clients not configured to use the proxy will bypass it entirely unless you otherwise block network access for unproxied traffic.

8 Chapter 2. Getting Started Apache-Trafﬁcserver-Server Documentation, latest

With transparent proxying, clients require no configuration and may be given no option to bypass the proxy. This configuration requires your network to route all requests automatically to the proxy. The details of how to accomplish this routing are entirely dependent upon the layout of your network and the routing devices you use. For a more detailed discussion of the options, and starting points for configuring each in your network environment, please refer to the Forward Proxy section of the Administrator’s Guide.

Logging and Monitoring

Conﬁguring Log Output

The log formats used by Traffic Server are highly configurable. While this guide will not go into full detail of this versatility, it is useful to consider what style of logging you would like to perform. If your organization already makes use of log monitoring or analysis tools that understand, for example, Netscape Extended-2 format you may wish to enable that logging format in addition to, or instead of, the default Traffic Server logs. The Administrator’s Guide discusses logging options in great detail in Logging.

Using Trafﬁc Top

Using Stats Over HTTP

Using Cache Inspector

Further Steps

By this point, you should have a fully functioning caching proxy, whether to help improve the customer facing performance of your website, or to assist in speeding up Internet access for your office while allowing for the possibility of access control, content filtering, and/or usage monitoring. However, it is quite likely that your installation is not yet tuned properly and may not be providing the best experience Apache Traffic Server™ has to offer. It is strongly recommended that you consult the Performance Tuning guide. You may also want to learn more about Event and Error Monitoring, or ensuring that your installation is properly secured by reading the Security section. Properly sizing your cache, both the on-disk cache and the companion memory cache, are important topics covered in.

2.4. Logging and Monitoring 9 Apache-Trafﬁcserver-Server Documentation, latest

10 Chapter 2. Getting Started CHAPTER 3

Administrator’s Guide

Global data networking has become part of everyday life: Internet users request billions of documents and petabytes of data, on a daily basis, to and from all parts of the world. Information is free, abundant, and accessible. Unfortunately, global data networking can also be a nightmare for IT professionals as they struggle with overloaded servers and congested networks. It can be challenging to consistently and reliably accommodate society’s growing data demands. Traffic Server is a high-performance web proxy cache that improves network efficiency and performance by caching frequently-accessed information at the edge of the network. This brings content physically closer to end users, while enabling faster delivery and reduced bandwidth use. Traffic Server is designed to improve content delivery for enterprises, Internet service providers (ISPs), backbone providers, and large intranets by maximizing existing and available bandwidth.

Trafﬁc Server Deployment Options

To best suit your needs, Traffic Server can be deployed in several ways: • WEB • • The following sections provide a summary of these Traffic Server deployment options. Please keep in mind that with every one of these options Traffic Server can be run as a single instance, or as a multi-node cluster.

Trafﬁc Server WEB

As a web proxy cache, Trafﬁc Server receives user requests for web content as those requests travel to the destined web server (origin server). If Trafﬁc Server contains the requested content, then it serves the content directly. If the

11 Apache-Trafﬁcserver-Server Documentation, latest

requested content is not available from cache, then Traffic Server acts as a proxy: it obtains the content from the origin server on the user’s behalf and also keeps a copy to satisfy future requests. Traffic Server provides explicit proxy caching, in which the user’s client software must be configured to send requests directly to Traffic Server. Explicit proxy caching is described in the Explicit Proxy Caching chapter. Traffic Server can also be employed as a transparent caching proxy server, in which the client software needs no special configuration or even knowledge of the proxy’s existence. This setup is described in the Transparent Proxying section.

Trafﬁc Server

As a reverse proxy, Traffic Server is configured to be the origin server to which the user is trying to connect (typically, the origin server’s advertised hostname resolves to Traffic Server, which acts as the real origin server). The reverse proxy feature is also called server acceleration. Reverse proxy is described in more detail in Reverse Proxy and HTTP Redirects.

Trafﬁc Server

Traffic Server can participate in flexible cache hierarchies, in which Internet requests not fulfilled from one cache are routed to other regional caches, thereby leveraging the contents and proximity of nearby caches. In a hierarchy of proxy servers, Traffic Server can act either as a parent or a child cache to other Traffic Server systems or to similar caching products. Traffic Server supports ICP (Internet Cache Protocol) peering. Hierarchical caching is described in more detail in Hierarchical Caching.

There are a number of deployment options that Trafﬁc Server does not support right out of the box. Such functionality may be implemented in a plugin, but in some cases Trafﬁc Server’s internal APIs or architectural restrictions won’t make it easy: • Load Balancing - note that there is an experimental plugin for this, Balancer Plugin.

Trafﬁc Server Components

Trafﬁc Server consists of several components that work together to form a web proxy cache you can easily monitor and conﬁgure.

The Trafﬁc Server Cache

The Traffic Server cache consists of a high-speed object database called the object store. The object store indexes objects according to URLs and associated headers. Using sophisticated object management, the object store can cache alternate versions of the same object (perhaps in a different language or encoding type). It can also efficiently store very small and very large objects, thereby minimizing wasted space. When the cache is full, Traffic Server removes stale data to ensure that the most requested objects are readily available and fresh. Traffic Server is designed to tolerate total disk failures on any of the cache disks. If the disk fails completely, then Traffic Server marks the entire disk as corrupt and continues to use remaining disks. If all of the cache disks fail, then Traffic Server switches to proxy-only mode. You can partition the cache to reserve a certain amount of disk space for storing data for specific protocols and origin servers. For more information about the cache, see HTTP Proxy Caching.

12 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

RAM

Traffic Server maintains a small RAM cache that contains extremely popular objects. This RAM cache serves the most popular objects as fast as possible and reduces load on disks, especially during temporary traffic peaks. You can configure the RAM cache size to suit your needs. For detailed information, refer to Changing the Size of the RAM Cache.

Host

The Traffic Server host database stores the domain name server (DNS) entries of origin servers to which Traffic Server connects to fulfill user requests. This information is used to adapt future protocol interactions and optimize performance. Along with other information, the host database tracks: • DNS information (for fast conversion of hostnames to IP addresses). • The HTTP version of each host (so advanced protocol features can be used with hosts running modern servers). • Host reliability and availability information (so users will not wait for servers that are not running).

DNS

Traffic Server includes a fast, asynchronous DNS resolver to streamline conversion of hostnames to IP addresses. Traffic Server implements the DNS resolver natively by directly issuing DNS command packets rather than relying on slower, conventional resolver libraries. Since many DNS queries can be issued in parallel and a fast DNS cache maintains popular bindings in memory, DNS traffic is reduced.

Trafﬁc Server Processes

Traffic Server contains three processes that work together to serve requests and manage, control, and monitor the health of the system. • The traffic_server process is the transaction processing engine of Traffic Server. It is responsible for accepting connections, processing protocol requests, and serving documents from the cache or origin server. • The traffic_manager process is the command and control facility of the Traffic Server, responsible for launching, monitoring, and reconfiguring the traffic_server process. The traffic_manager process is also responsible for the proxy autoconfiguration port, the statistics interface, cluster administration, and virtual IP failover. If the traffic_manager process detects a traffic_server process failure, it instantly restarts the process but also maintains a connection queue of all incoming requests. All incoming connections that arrive in the several seconds before full server restart are saved in the connection queue and processed in first-come, first-served order. This connection queueing shields users from any server restart downtime. • The traffic_cop process monitors the health of both the traffic_server and traffic_manager processes. The traffic_cop process periodically (several times each minute) queries the traffic_server and traffic_manager processes by issuing heartbeat requests to fetch synthetic web pages. In the event of failure (if no response is received within a timeout interval or if an incorrect response is received), traffic_cop restarts the traffic_manager and traffic_server processes. The figure below illustrates the three Traffic Server processes.

3.1. 13 Apache-Trafﬁcserver-Server Documentation, latest

3.1: Illustration of the three Trafﬁc Server Processes

14 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Traffic Server offers the following administration options: • The traffic_ctl command-line interface is a text-based interface from which you can monitor Traffic Server performance and network traffic, as well as configure the Traffic Server system. From Traffic Line, you can execute individual commands or script a series of commands in a shell. • Various configuration files enable you to configure Traffic Server through a simple file-editing and signal- handling interface. Any changes you make through Traffic Line is automatically made to the configuration files as well. • Finally, there is a clean C API which can be put to good use from a multitude of languages. The Traffic Server Admin Client demonstrates this for Perl.

Trafﬁc Analysis Options

Traffic Server provides several options for network traffic analysis and monitoring: • traffic_ctl enables you to collect and process statistics obtained from network traffic information. • Transaction logging enables you to record information (in a log file) about every request Traffic Server receives and every error it detects. By analyzing the log files, you can determine how many clients used the Traffic Server cache, how much information each of them requested, and what pages were most popular. You can also see why a particular transaction was in error and what state the Traffic Server was in at a particular time. For example, you can see that Traffic Server was restarted or that cluster communication timed out. Traffic Server supports several standard log file formats, such as Squid and Netscape, and its own custom format. You can analyze the standard format log files with off-the-shelf analysis packages. To help with log file analysis, you can separate log files so that they contain information specific to protocol or hosts. Traffic Server event and error logging, monitoring, and analysis is covered in greater detail in Event and Error Moni- toring.

Trafﬁc Server

Traffic Server provides numerous options that enable you to establish secure communication between the Traffic Server system and other computers on the network. Using the security options, you can do the following: • Control client access to the Traffic Server proxy cache. • Configure Traffic Server to use multiple DNS servers to match your site’s security configuration. For example, Traffic Server can use different DNS servers, depending on whether it needs to resolve hostnames located inside or outside a firewall. This enables you to keep your internal network configuration secure while continuing to provide transparent access to external sites on the Internet. • Configure Traffic Server to verify that clients are authenticated before they can access content from the Traffic Server cache. • Secure connections in reverse proxy mode between a client and Traffic Server, and Traffic Server and the origin server, using the SSL termination option. • Control access via SSL (Secure Sockets Layer). Traffic Server security options are described in more detail in Security.

3.1. 15 Apache-Trafﬁcserver-Server Documentation, latest

Tuning Trafﬁc Server

Finally, this last chapter on Performance Tuning discusses the vast number of options that allow administrators to optimally tune Apache Trafﬁc Server for maximum performance.

Trafﬁc Server

Supported Platforms

Trafﬁc Server Versioning

Before you get started with Trafﬁc Server you may have to decide which version you want to use. Trafﬁc Server follows the Semantic Versioning guidelines. A complete version number is made of a version-triplet: MAJOR.MINOR.PATCH. As of v4.0.0, there are no longer any development (or unstable) releases. All releases are considered stable and ready for production use. Releases within a major version are always upgrade compatible. More details are available on the New Release Processes wiki page. Sometimes we speak of trunk, master or HEAD, all of which are used interchangeably. Trunk and master, or sometimes TIP or HEAD, refer to the latest code in a Git version control system (also referred to as a repository or Git repo). Master is always kept releasable, and compatible with the current major release version. Incompatible changes are sometimes committed on a next-major release branch; for example, we have the 5.0.x branch where changes incompatible with 4.x are managed.

Binary Distributions

Building From Source

Retrieving Tarballs

Compressed archives of the source code for Traffic Server are available on the official website’s Downloads page. From there you may select the version most appropriate for your needs. The Traffic Server project does not provide binary downloads.

Cloning from Version Control

Trafﬁc Server uses a public Git repository for version control. This repository will also provide the most cutting edge source code if you wish to test out the latest features and bug ﬁxes.

: We do also have a GitHub Mirror that you may use to submit pull requests. However, it may not be entirely up-to-date, and you should always refer to our ofﬁcial project Git repository for the very latest state of the source code.

Build Dependencies

In order to build Trafﬁc Server from source you will need the following development tools and libraries installed: • pkgconﬁg

16 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

• libtool • gcc (>= 4.3 or clang > 3.0) • GNU make • openssl • tcl • expat • pcre • libcap • ﬂex (for TPROXY) • hwloc • lua • curses (for traffic_top) • curl (for traffic_top) If you’re building from a git clone, you’ll also need: • git • autoconf • automake

Layouts

Preparing the Source Tree

If you are buiding from a checkout of the Git repository, you will need to prepare the source tree by regenerating the conﬁguration scripts. This is performed by running: autoreconf- if

At the base directory of your local clone.

Conﬁguration Options

Trafﬁc Server uses the standard configure script method of conﬁguring the source tree for building. A full list of available options may always be obtained by running the following in the base directory of your unpackaged archive or Git working copy:

./configure--help

1.A configure script will be generated from configure.ac which may now be used to conﬁgure the source tree for your build.

./configure--prefix=/opt/ats

By default, Trafﬁc Server will be built to use the nobody user and group. You may change this with the --with-user argument to configure:

3.2. Trafﬁc Server 17 Apache-Trafﬁcserver-Server Documentation, latest

./configure--prefix=/opt/ats-- with-user=tserver

If dependencies are not in standard paths (/usr/local or /usr), you may need to pass options to configure to account for that:

./configure--prefix=/opt/ats-- with-lua=/opt/csw

Most configure path-options accept a format of “INCLUDE_PATH:LIBRARY_PATH”:

./configure--prefix=/opt/ats-- with-pcre=/opt/csw/include:/opt/csw/lib/amd64

2. Once the source tree has been conﬁgured, you may proceed on to building with the generated Makeﬁles. The make check command may be used to perform sanity checks on the resulting build, prior to installation, and it is recommended that you use this.

make make check

3. With the source built and checked, you may now install all of the binaries, header ﬁles, documentation, and other artifacts to their ﬁnal locations on your system.

sudo make install

4. Finally, it is recommended that you run the regression test suite. Please note that the regression tests will only be successful with the default layout.

cd/opt/ats sudo bin/traffic_server-R1

Adding SPDY Support

Trafﬁc Server v5.0.x and above are capable of supporting SPDY, but it is optional and must be explicitly conﬁgured during the build process. The support of SPDY is deprecated as of ATS v6.2.0, and will be removed in v7.0.0. 1. Clone the spdylay Git repository from tatsuhiro.

git clone https://github.com/tatsuhiro-t/spdylay

2. Generate fresh conﬁgure scripts.

autoreconf- if

3. Run the generated conﬁguration script, choosing an installation location for the overlay.

./configure--prefix=/opt/spdylay

4. Build and install the overlay.

make install

5. Set the PKG_CONFIG_PATH enviroment variable to the appropriate path in your newly-installed overlay directory.

export PKG_CONFIG_PATH=/opt/spdylay/lib/pkgconfig/

18 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

6. Finally, you may proceed with the regular building of Trafﬁc Server from source, with the addition of a single new conﬁguration option to enable SPDY support in the produced binaries.

./configure--enable-spdy

Start Trafﬁc Server

To start Traffic Server manually, issue the trafficserver command, passing in the subcommand start. This command starts all the processes that work together to process Traffic Server requests as well as manage, control, and monitor the health of the Traffic Server system. bin/trafficserver start

The traffic_ctl provides a quick way of viewing Traffic Server statistics and configuring the Traffic Server system via a command-line interface. traffic_ctl commands take the following form: bin/traffic_ctl COMMAND COMMAND...

For a list of traffic_ctl commands, enter: bin/traffic_ctl

Please note that traffic_ctl, while a ﬁne tool for administrators, is a poor choice for automation, especially that of monitoring. See our chapter on monitoring for how to do that more efﬁciently and effectively.

Stop Trafﬁc Server

To stop Trafﬁc Server, always use the trafficserver command, passing in the attribute stop. This command stops all the Trafﬁc Server processes (traffic_manager, traffic_server, and traffic_cop). Do not manually stop processes, as this can lead to unpredictable results. bin/trafficserver stop

Session Protocol

Traffic Server supports some session level protocols in place of or on top of HTTP. These can be provided by a plugin (see New Protocol Plugins) or be one that is supported directly by Traffic Server. Note that the SPDY protocol is deprecated as of v6.2.0, and will be removed in v7.0.0. Session protocols are specified by explicit names: • http/0.9 • http/1.0 • http/1.1 • http/2 • spdy/1

3.3. 19 Apache-Trafﬁcserver-Server Documentation, latest

• spdy/2 • spdy/3 • spdy/3.1 The session protocols supported on a proxy port are a subset of these values. For convenience some pseudo-values are defined in terms of these fundamental protocols: • http means http/0.9, http/1.0, and http/1.1 • spdy means spdy/3 and spdy/3.1. • http2 means http/2 Each proxy port can be configured in :ts:cv:‘proxy.config.http.server_ports‘ to support a subset of these session protocols. For TLS enabled connections this configuration controls which protocols are offered by NPN. Protocol sniffing is use for non-TLS proxy ports to determine which protocol is being used by the client. If the detected protocol is not supported for that proxy port the connection is dropped.

HTTP Proxy Caching

HTTP proxy caching enables you to store copies of frequently-accessed web objects (such as documents, images, and articles) and then serve this information to users on demand. It improves performance and frees up Internet bandwidth for other tasks.

Understanding HTTP Web Proxy Caching

Internet users direct their requests to web servers all over the Internet. A caching server must act as a web proxy server so it can serve those requests. After a web proxy server receives requests for web objects, it either serves the requests or forwards them to the origin server (the web server that contains the original copy of the requested information). The Traffic Server proxy supports explicit proxy caching, in which the user’s client software must be configured to send requests directly to the Traffic Server proxy. The following overview illustrates how Traffic Server serves a request. 1. Traffic Server receives a client request for a web object. 2. Using the object address, Traffic Server tries to locate the requested object in its object database (cache). 3. If the object is in the cache, then Traffic Server checks to see if the object is fresh enough to serve. If it is fresh, then Traffic Server serves it to the client as a cache hit (see the figure below). 4. If the data in the cache is stale, then Traffic Server connects to the origin server and checks if the object is still fresh (a revalidation). If it is, then Traffic Server immediately sends the cached copy to the client. 5. If the object is not in the cache (a cache miss) or if the server indicates the cached copy is no longer valid, then Traffic Server obtains the object from the origin server. The object is then simultaneously streamed to the client and the Traffic Server local cache (see the figure below). Subsequent requests for the object can be served faster because the object is retrieved directly from cache. Caching is typically more complex than the preceding overview suggests. In particular, the overview does not discuss how Traffic Server ensures freshness, serves correct HTTP alternates, and treats requests for objects that cannot or should not be cached. The following sections discuss these issues in greater detail.

Ensuring Cached Object Freshness

When Traffic Server receives a request for a web object, it first tries to locate the requested object in its cache. If the object is in cache, then Traffic Server checks to see if the object is fresh enough to serve. For HTTP objects, Traffic Server supports optional author-specified expiration dates. Traffic Server adheres to these expiration dates; otherwise,

20 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

3.2: A cache hit

3.3: A cache miss

3.3. 21 Apache-Trafﬁcserver-Server Documentation, latest it picks an expiration date based on how frequently the object is changing and on administrator-chosen freshness guidelines. Objects can also be revalidated by checking with the origin server to see if an object is still fresh.

HTTP Object Freshness

Traffic Server determines whether an HTTP object in the cache is fresh by checking the following conditions in order: • Checking the Expires or max-age header Some HTTP objects contain Expires headers or max-age headers that explicitly define how long the object can be cached. Traffic Server compares the current time with the expiration time to determine if the object is still fresh. • Checking the Last-Modified / Date header If an HTTP object has no Expires header or max-age header, then Traffic Server can calculate a freshness limit using the following formula:

freshness_limit= ( date- last_modified ) * 0.10

where date is the date in the object’s server response header and last_modified is the date in the Last-Modified header. If there is no Last-Modified header, then Traffic Server uses the date the object was written to cache. The value 0.10 (10 percent) can be increased or reduced to better suit your needs. Refer to Modifying Aging Factor for Freshness Computations. The computed freshness limit is bound by a minimum and maximum value. Refer to Setting Absolute Freshness Limits for more information. • Checking the absolute freshness limit For HTTP objects that do not have Expires headers or do not have both Last-Modified and Date headers, Traffic Server uses a maximum and minimum freshness limit. Refer to Setting Absolute Freshness Limits. • Checking revalidate rules in cache.config Revalidate rules apply freshness limits to specific HTTP objects. You can set freshness limits for objects originating from particular domains or IP addresses, objects with URLs that contain specified regular expressions, objects requested by particular clients, and so on. Refer to cache.config.

Modifying Aging Factor for Freshness Computations

If an object does not contain any expiration information, then Traffic Server can estimate its freshness from the Last-Modified and Date headers. By default, Traffic Server stores an object for 10% of the time that elapsed since it last changed. You can increase or reduce the percentage according to your needs. To modify the aging factor for freshness computations: 1. Change the value for :ts:cv:‘proxy.config.http.cache.heuristic_lm_factor‘. 2. Run the traffic_ctl config reload command to apply the configuration changes.

Setting Absolute Freshness Limits

Some objects do not have Expires headers or do not have both Last-Modified and Date headers. To control how long these objects are considered fresh in the cache, specify an absolute freshness limit. To specify an absolute freshness limit:

22 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

1. Edit the variables :ts:cv:‘proxy.config.http.cache.heuristic_min_lifetime‘ and :ts:cv:‘proxy.config.http.cache.heuristic_max_lifetime‘ in records.config. 2. Run the traffic_ctl config reload command to apply the configuration changes.

Specifying Header Requirements

To further ensure freshness of the objects in the cache, configure Traffic Server to cache only objects with specific headers. By default, Traffic Server caches all objects (including objects with no headers); you should change the default setting only for specialized proxy situations. If you configure Traffic Server to cache only HTTP objects with Expires or max-age headers, then the cache hit rate will be noticeably reduced (since very few objects will have explicit expiration information). To configure Traffic Server to cache objects with specific headers: 1. Change the value for :ts:cv:‘proxy.config.http.cache.required_headers‘ in records.config. 2. Run the traffic_ctl config reload command to apply the configuration changes.

Cache-Control Headers

Even though an object might be fresh in the cache, clients or servers often impose their own constraints that preclude retrieval of the object from the cache. For example, a client might request that a object not be retrieved from a cache, or if it does allow cache retrieval, then it cannot have been cached for more than 10 minutes. Traffic Server bases the servability of a cached object on Cache-Control headers that appear in both client requests and server responses. The following Cache-Control headers affect whether objects are served from cache: • The no-cache header, sent by clients, tells Traffic Server that it should not serve any objects directly from the cache. When present in a client request, Traffic Server will always obtain the object from the origin server. You can configure Traffic Server to ignore client no-cache headers. Refer to Configuring Traffic Server to Ignore Client no-cache Headers for more information. • The max-age header, sent by servers, is compared to the object age. If the age is less than max-age, then the object is fresh and can be served from the Traffic Server cache. • The min-fresh header, sent by clients, is an acceptable freshness tolerance. This means that the client wants the object to be at least this fresh. Unless a cached object remains fresh at least this long in the future, it is revalidated. • The max-stale header, sent by clients, permits Traffic Server to serve stale objects provided they are not too old. Some browsers might be willing to take slightly stale objects in exchange for improved performance, especially during periods of poor Internet availability. Traffic Server applies Cache-Control servability criteria after HTTP freshness criteria. For example, an object might be considered fresh but will not be served if its age is greater than its max-age.

Revalidating HTTP Objects

When a client requests an HTTP object that is stale in the cache, Traffic Server revalidates the object. A revalidation is a query to the origin server to check if the object is unchanged. The result of a revalidation is one of the following: • If the object is still fresh, then Traffic Server resets its freshness limit and serves the object. • If a new copy of the object is available, then Traffic Server caches the new object (thereby replacing the stale copy) and simultaneously serves the object to the client.

3.3. 23 Apache-Trafﬁcserver-Server Documentation, latest

• If the object no longer exists on the origin server, then Traffic Server does not serve the cached copy. • If the origin server does not respond to the revalidation query, then Traffic Server serves the stale object along with a 111 Revalidation Failed warning. By default, Traffic Server revalidates a requested HTTP object in the cache if it considers the object to be stale. Traffic Server evaluates object freshness as described in HTTP Object Freshness. You can reconfigure how Traffic Server evaluates freshness by selecting one of the following options: Traffic Server considers all HTTP objects in the cache to be stale: Always revalidate HTTP objects in the cache with the origin server. Traffic Server considers all HTTP objects in the cache to be fresh: Never revalidate HTTP objects in the cache with the origin server. Traffic Server considers all HTTP objects without Expires or Cache-control headers to be stale: Revalidate all HTTP objects without Expires or Cache-Control headers. To configure how Traffic Server revalidates objects in the cache, you can set specific revalidation rules in cache. config. To configure revalidation options 1. Edit the variable :ts:cv:‘proxy.config.http.cache.when_to_revalidate‘ in records.config. 2. Run the traffic_ctl config reload command to apply the configuration changes.

Pushing Content into the Cache

Trafﬁc Server supports the HTTP PUSH method of content delivery. Using HTTP PUSH, you can deliver content directly into the cache without client requests.

Conﬁguring Trafﬁc Server for PUSH Requests

Before you can deliver content into your cache using HTTP PUSH, you must configure Traffic Server to accept PUSH requests. 1. Edit ip_allow.config to allow PUSH from the appropriate addresses. 2. Update :ts:cv:‘proxy.config.http.push_method_enabled‘ in records.config:

CONFIG proxy.config.http.push_method_enabled INT1

3. Run the command traffic_ctl config reload to apply the conﬁguration changes.

Understanding HTTP PUSH

PUSH uses the HTTP 1.1 message format. The body of a PUSH request contains the response header and response body that you want to place in the cache. The following is an example of a PUSH request:

PUSH http://www.company.com HTTP/1.0 Content-length: 84

HTTP/1.0 200OK Content-type: text/html Content-length: 17

24 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

: Your PUSH headers must include Content-length, the value for which must include both headers and body byte counts. The value is not optional, and an improper (too large or too small) value will result in undesirable behavior.

Tools that will help manage pushing

Trafﬁc Server comes with a Perl script for pushing, tspush, which can assist with understanding how to write scripts for pushing content yourself.

Pinning Content in the Cache

The Cache Pinning Option configures Traffic Server to keep certain HTTP objects in the cache for a specified time. You can use this option to ensure that the most popular objects are in cache when needed and to prevent Traffic Server from deleting important objects. Traffic Server observes Cache-Control headers and pins an object in the cache only if it is indeed cacheable. To set cache pinning rules: 1. Enable :ts:cv:‘proxy.config.cache.permit.pinning‘ in records.config:

CONFIG proxy.config.cache.permit.pinning INT1

2. Add a rule in cache.config for each URL you want Trafﬁc Server to pin in the cache. For example:

url_regex=^https?://(www.)?apache.org/dev/ pin-in-cache=12h

3. Run the command traffic_ctl config reload to apply the conﬁguration changes.

Caching HTTP Objects

When Traffic Server receives a request for a web object that is not in the cache, it retrieves the object from the origin server and serves it to the client. At the same time, Traffic Server checks if the object is cacheable before storing it in its cache to serve future requests. Traffic Server responds to caching directives from clients and origin servers, as well as directives you specify through configuration options and files.

Client Directives

By default, Traffic Server does not cache objects with the following request headers: • Authorization • Cache-Control: no-store • Cache-Control: no-cache To configure Traffic Server to ignore this request header, refer to Configuring Traffic Server to Ignore Client no-cache Headers.

3.3. 25 Apache-Trafﬁcserver-Server Documentation, latest

• Cookie (for text objects) By default, Traffic Server caches objects served in response to requests that contain cookies (even if the object is text). You can configure Traffic Server to not cache cookied content of any type, cache all cookied content, or cache cookied content that is of image type only. For more information, refer to Caching Cookied Objects.

Conﬁguring Trafﬁc Server to Ignore Client no-cache Headers

By default, Traffic Server strictly observes client Cache-Control: no-cache directives. If a requested object contains a no-cache header, then Traffic Server forwards the request to the origin server even if it has a fresh copy in cache. You can configure Traffic Server to ignore client no-cache directives such that it ignores no-cache headers from client requests and serves the object from its cache. 1. Edit :ts:cv:‘proxy.config.http.cache.ignore_client_no_cache‘ in records.config.

CONFIG proxy.config.http.cache.ignore_client_no_cache INT1

2. Run the command traffic_ctl config reload to apply the conﬁguration changes.

Origin Server Directives

By default, Traffic Server does not cache objects with the following response headers: • Cache-Control: no-store • Cache-Control: private • WWW-Authenticate To configure Traffic Server to ignore WWW-Authenticate headers, refer to Configuring Traffic Server to Ignore WWW-Authenticate Headers. • Set-Cookie • Cache-Control: no-cache To configure Traffic Server to ignore no-cache headers, refer to Configuring Traffic Server to Ignore Server no-cache Headers. • Expires header with a value of 0 (zero) or a past date.

Conﬁguring Trafﬁc Server to Ignore Server no-cache Headers

By default, Traffic Server strictly observes Cache-Control: no-cache directives. A response from an origin server with a no-cache header is not stored in the cache and any previous copy of the object in the cache is removed. If you configure Traffic Server to ignore no-cache headers, then Traffic Server also ignores no-store headers. The default behavior of observing no-cache directives is appropriate in most cases. To configure Traffic Server to ignore server no-cache headers: 1. Edit :ts:cv:‘proxy.config.http.cache.ignore_server_no_cache‘ in records.config.

CONFIG proxy.config.http.cache.ignore_server_no_cache INT1

2. Run the command traffic_ctl config reload to apply the conﬁguration changes.

26 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Conﬁguring Trafﬁc Server to Ignore WWW-Authenticate Headers

By default, Traffic Server does not cache objects that contain WWW-Authenticate response headers. The WWW-Authenticate header contains authentication parameters the client uses when preparing the authentication challenge response to an origin server. When you configure Traffic Server to ignore origin server WWW-Authenticate headers, all objects with WWW-Authenticate headers are stored in the cache for future requests. However, the default behavior of not caching objects with WWW-Authenticate headers is appropriate in most cases. Only configure Traffic Server to ignore server WWW-Authenticate headers if you are knowledgeable about HTTP 1.1. To configure Traffic Server to ignore server WWW-Authenticate headers: 1. Edit :ts:cv:‘proxy.config.http.cache.ignore_authentication‘ in records.config.

CONFIG proxy.config.http.cache.ignore_authentication INT1

2. Run the command traffic_ctl config reload to apply the conﬁguration changes.

Conﬁguration Directives

In addition to client and origin server directives, Traffic Server responds to directives you specify through configuration options and files. You can configure Traffic Server to do the following: • Not cache any HTTP objects. Refer to Disabling HTTP Object Caching. • Cache dynamic content. That is, objects with URLs that end in .asp or contain a question mark (?), semicolon (;), or cgi. For more information, refer to Caching Dynamic Content. • Cache objects served in response to the Cookie: header. Refer to Caching Cookied Objects. • Observe never-cache rules in cache.config.

Disabling HTTP Object Caching

By default, Trafﬁc Server caches all HTTP objects except those for which you have set never-cache as action rules in cache.config. You can disable HTTP object caching so that all HTTP objects are served directly from the origin server and never cached, as detailed below. To disable HTTP object caching manually: 1. Set :ts:cv:‘proxy.conﬁg.http.cache.http‘ to 0 in records.config.

CONFIG proxy.config.http.cache.http INT0

2. Run the command traffic_ctl config reload to apply the conﬁguration changes.

Caching Dynamic Content

A URL is considered dynamic if it ends in .asp or contains a question mark (?), a semicolon (;), or cgi. By default, Traffic Server caches dynamic content. You can configure the system to ignore dynamic looking content, although this is recommended only if the content is truly dynamic, but fails to advertise so with appropriate Cache-Control headers. To configure Traffic Server’s cache behaviour in regard to dynamic content:

3.3. 27 Apache-Trafﬁcserver-Server Documentation, latest

1. Edit :ts:cv:‘proxy.conﬁg.http.cache.cache_urls_that_look_dynamic‘ in records.config. To disable caching, set the variable to 0, and to explicitly permit caching use 1.

CONFIG proxy.config.http.cache.cache_urls_that_look_dynamic INT0

2. Run the command traffic_ctl config reload to apply the conﬁguration changes.

Caching Cookied Objects

By default, Traffic Server caches objects served in response to requests that contain cookies. This is true for all types of objects including text. Traffic Server does not cache cookied text content because object headers are stored along with the object, and personalized cookie header values could be saved with the object. With non-text objects, it is unlikely that personalized headers are delivered or used. You can reconfigure Traffic Server to: • Not cache cookied content of any type. • Cache cookied content that is of image type only. • Cache all cookied content regardless of type. To configure how Traffic Server caches cookied content: 1. Edit :ts:cv:‘proxy.config.http.cache.cache_responses_to_cookies‘ in records.config. 2. Run the command traffic_ctl config reload to apply the configuration changes.

Forcing Object Caching

You can force Traffic Server to cache specific URLs (including dynamic URLs) for a specified duration, regardless of Cache-Control response headers. To force document caching: 1. Add a rule for each URL you want Traffic Server to pin to the cache cache.config:

url_regex=^https?://(www.)?apache.org/dev/ ttl-in-cache=6h

2. Run the command traffic_ctl config reload to apply the conﬁguration changes.

Caching HTTP Alternates

Some origin servers answer requests to the same URL with a variety of objects. The content of these objects can vary widely, according to whether a server delivers content for different languages, targets different browsers with different presentation styles, or provides different document formats (HTML, XML). Different versions of the same object are termed alternates and are cached by Traffic Server based on Vary response headers. You can specify additional request and response headers for specific Content-Type values that Traffic Server will identify as alternates for caching. You can also limit the number of alternate versions of an object allowed in the cache.

Conﬁguring How Trafﬁc Server Caches Alternates

To configure how Traffic Server caches alternates: 1. Edit the following variables in records.config: • :ts:cv:‘proxy.config.http.cache.enable_default_vary_headers‘

28 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

• :ts:cv:‘proxy.config.http.cache.vary_default_text‘ • :ts:cv:‘proxy.config.http.cache.vary_default_images‘ • :ts:cv:‘proxy.config.http.cache.vary_default_other‘ 2. Run the command traffic_ctl config reload to apply the configuration changes.

: If you specify Cookie as the header ﬁeld on which to vary in the above variables, make sure that the variable :ts:cv:‘proxy.conﬁg.http.cache.cache_responses_to_cookies‘ is set appropriately.

Limiting the Number of Alternates for an Object

You can limit the number of alternates Trafﬁc Server can cache per object (the default is 3).

: Large numbers of alternates can affect Trafﬁc Server cache performance because all alternates have the same URL. Although Trafﬁc Server can look up the URL in the index very quickly, it must scan sequentially through available alternates in the object store.

To alter the limit on the number of alternates: 1. Edit :ts:cv:‘proxy.conﬁg.cache.limits.http.max_alts‘ in records.config.

CONFIG proxy.config.cache.limits.http.max_alts INT5

2. Run the command traffic_ctl config reload to apply the conﬁguration changes.

Using Congestion Control

The Congestion Control option enables you to configure Traffic Server to stop forwarding HTTP requests to origin servers when they become congested. Traffic Server then sends the client a message to retry the congested origin server later. To enable this option: 1. Set :ts:cv:‘proxy.config.http.congestion_control.enabled‘ to 1 in records.config.

CONFIG proxy.config.http.congestion_control.enabled INT1

2. Create rules in congestion.config to specify: • Which origin servers Traffic Server tracks for congestion. • The timeouts Traffic Server uses, depending on whether a server is congested. • The page Traffic Server sends to the client when a server becomes congested. • Whether Traffic Server tracks the origin servers by IP address or by hostname. 3. Run the command traffic_ctl config reload to apply the configuration changes.

3.3. 29 Apache-Trafﬁcserver-Server Documentation, latest

Using Transaction Buffering Control

By default, I/O operations are run at full speed, as fast as either Traffic Server, the network, or the cache can support. This can be problematic for large objects if the client side connection is significantly slower. In such cases the content will be buffered in ram while waiting to be sent to the client. This could potentially also happen for POST requests if the client connection is fast and the origin server connection slow. If very large objects are being used this can cause the memory usage of Traffic Server to become very large. This problem can be ameliorated by controlling the amount of buffer space used by a transaction. A high water and low water mark are set in terms of bytes used by the transaction. If the buffer space in use exceeds the high water mark, the connection is throttled to prevent additional external data from arriving. Internal operations continue to proceed at full speed until the buffer space in use drops below the low water mark and external data I/O is re-enabled. Although this is intended primarily to limit the memory usage of Traffic Server it can also serve as a crude rate limiter by setting a buffer limit and then throttling the client side connection either externally or via a transform. This will cause the connection to the origin server to be limited to roughly the client side connection speed. Traffic Server does network I/O in large chunks (32K or so) and therefore the granularity of transaction buffering control is limited to a similar precision. The buffer size calculations include all elements in the transaction, including any buffers associated with transform plugins. Transaction buffering control can be enabled globally by using configuration variables or by TSHttpTxnConfigIntSet() in a plugin. Value Variable TSHttpTxnConfigIntSet() key Enable TS_CONFIG_HTTP_FLOW_CONTROL_ENABLED :ts:cv:‘proxy.config.http.flow_control.enabled‘ buffering Set high water TS_CONFIG_HTTP_FLOW_CONTROL_HIGH_WATER :ts:cv:‘proxy.config.http.flow_control.high_water‘ Set low water TS_CONFIG_HTTP_FLOW_CONTROL_LOW_WATER :ts:cv:‘proxy.config.http.flow_control.low_water‘ Be careful to always have the low water mark equal or less than the high water mark. If you set only one, the other will be set to the same value. If using TSHttpTxnConfigIntSet(), it must be called no later than TS_HTTP_READ_RESPONSE_HDR_HOOK.

Reducing Origin Server Requests (Avoiding the Thundering Herd)

When an object can not be served from cache, the request will be proxied to the origin server. For a popular object, this can result in many near simultaneous requests to the origin server, potentially overwhelming it or associated resources. There are several features in Trafﬁc Server that can be used to avoid this scenario.

Read While Writer

When Traffic Server goes to fetch something from origin, and upon receiving the response, any number of clients can be allowed to start serving the partially filled cache object once background_fill_completed_threshold % of the object has been received. While some other HTTP proxies permit clients to begin reading the response immediately upon the proxy receiving data from the origin server, ATS does not begin allowing clients to read until after the complete HTTP response headers have been read and processed. This is a side-effect of ATS making no distinction between a cache refresh and a cold cache, which prevents knowing whether a response is going to be cacheable.

30 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

As non-cacheable responses from an origin server are generally due to that content being unique to different client requests, ATS will not enable read-while-writer functionality until it has determined that it will be able to cache the object. The following settings must be made in records.config to enable read-while-writer functionality in ATS:

CONFIG proxy.config.cache.enable_read_while_writer INT1 CONFIG proxy.config.http.background_fill_active_timeout INT0 CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000 CONFIG proxy.config.cache.max_doc_size INT0

All four configurations are required, for the following reasons: • :ts:cv:‘proxy.config.cache.enable_read_while_writer‘ being set to 1 turns the feature on, as it is off (0) by default. • The background fill feature (both :ts:cv:‘proxy.config.http.background_fill_active_timeout‘ and :ts:cv:‘proxy.config.http.background_fill_completed_threshold‘) should be allowed to kick in for every possible request. This is necessary in the event the writer (the first client session to request the object, which triggered ATS to contact the origin server) goes away. Another client session needs to take over the writer. As such, you should set the background fill timeouts and threshold to zero; this assures they never time out and are always allowed to kick in. • The :ts:cv:‘proxy.config.cache.max_doc_size‘ should be unlimited (set to 0), since the object size may be unknown, and going over this limit would cause a disconnect on the objects being served. Once these are enabled, you have something that is very close, but not quite the same, to Squid’s Collapsed Forwarding. In addition to the above settings, the settings :ts:cv:‘proxy.config.cache.read_while_writer.max_retries‘ and :ts:cv:‘proxy.config.cache.read_while_writer_retry.delay‘ allow to control the number of retries TS attempts to trigger read-while-writer until the download of first fragment of the object is completed:

CONFIG proxy.config.cache.read_while_writer.max_retries INT 10

CONFIG proxy.config.cache.read_while_writer_retry.delay INT 50

Fuzzy Revalidation

: These options are deprecated as of v6.2.0.

Trafﬁc Server can be set to attempt to revalidate an object before it becomes stale in cache. records.config contains the settings:

CONFIG proxy.config.http.cache.fuzz.time INT 240 CONFIG proxy.config.http.cache.fuzz.min_time INT0 CONFIG proxy.config.http.cache.fuzz.probability FLOAT 0.005

For every request for an object that occurs :ts:cv:‘proxy.conﬁg.http.cache.fuzz.time‘ before (in the example above, 240 seconds) the object is set to become stale, there is a small chance (:ts:cv:‘proxy.conﬁg.http.cache.fuzz.probability‘ == 0.5%) that the request will trigger a revalidation request to the origin.

: When revalidation occurs, the requested object is no longer available to be served from cache. Subsequent requests for that object will be proxied to the origin.

3.3. 31 Apache-Trafﬁcserver-Server Documentation, latest

For objects getting a few requests per second, these settings would offer a fairly low probability of revalidating the cached object before it becomes stale. This feature is not typically necessary at those rates, though, since odds are only one or a small number of connections would hit origin upon the objects going stale. Once request raise rise, the same fuzz.probability leads to a greater chance the object may be revalidated before becoming stale. This can prevent multiple clients simultaneously triggering contact with the origin server under higher loads, as they would do if no fuzziness was employed for revalidations. These settings are also overridable by remap rules and via plugins, so can be adjusted per request if necessary. Finally, :ts:cv:‘proxy.config.http.cache.fuzz.min_time‘ allows for different time periods to evaluate the probability of revalidation for small TTLs and large TTLs. Objects with small TTLs will start “rolling the revalidation dice” near the fuzz.min_time, while objects with large TTLs would start at fuzz.time. A logarithmic like function between determines the revalidation evaluation start time (which will be between fuzz. min_time and fuzz.time). As the object gets closer to expiring, the window start becomes more likely. By default this setting is not enabled, but should be enabled anytime you have objects with small TTLs. Note that this option predates overridable configurations, so you can achieve something similar with a plugin or remap.config settings. These configuration options are similar to Squid’s refresh_stale_hit configuration option.

Open Read Retry Timeout

The open read retry configurations attempt to reduce the number of concurrent requests to the origin for a given object. While an object is being fetched from the origin server, subsequent requests would wait :ts:cv:‘proxy.config.http.cache.open_read_retry_time‘ milliseconds before checking if the object can be served from cache. If the object is still being fetched, the subsequent requests will retry :ts:cv:‘proxy.config.http.cache.max_open_read_retries‘ times. Thus, subsequent requests may wait a total of (max_open_read_retries x open_read_retry_time) milliseconds before establishing an origin connection of its own. For instance, if they are set to 5 and 10 respectively, connections will wait up to 50ms for a response to come back from origin from a previous request, until this request is allowed through.

: These settings are inappropriate when objects are uncacheable. In those cases, requests for an object effectively become serialized. The subsequent requests would await at least open_read_retry_time milliseconds before being proxied to the origin.

It is advisable that this setting be used in conjunction with Read While Writer for big (those that take longer than (max_open_read_retries x open_read_retry_time) milliseconds to transfer) cacheable objects. With- out the read-while-writer settings enabled, while the initial fetch is ongoing, not only would subsequent requests be delayed by the maximum time, but also, those requests would result in unnecessary requests to the origin server. Since ATS now supports setting these settings per-request or remap rule, you can conﬁgure this to be suitable for your setup much more easily. The conﬁgurations are (with defaults):

CONFIG proxy.config.http.cache.max_open_read_retries INT-1 CONFIG proxy.config.http.cache.open_read_retry_time INT 10

The defaults are such that the feature is disabled and every connection is allowed to go to origin without artiﬁcial delay. When enabled, you will try max_open_read_retries times, each with an open_read_retry_time timeout.

32 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Open Write Fail Action

In addition to the open read retry settings TS supports a new setting :ts:cv:‘proxy.conﬁg.http.cache.open_write_fail_action‘ that allows to further reduce multiple concurrent requests hitting the origin for the same object by either returning a stale copy, in case of hit-stale or an error in case of cache miss for all but one of the requests.

Reverse Proxy and HTTP Redirects

As a reverse proxy cache, Traffic Server serves requests on behalf of origin servers. Traffic Server is configured in such a way that it appears to clients like a normal origin server.

Understanding Reverse Proxy Caching

With forward proxy caching, Traffic Server handles web requests to origin servers on behalf of the clients requesting the content. Reverse proxy caching (also known as server acceleration) is different because Traffic Server acts as a proxy cache on behalf of the origin servers that store the content. Traffic Server is configured to behave outwardly as origin server which the client is trying to connect to. In a typical scenario the advertised hostname of the origin server resolves to Traffic Server, which serves client requests directly, fetching content from the true origin server when necessary.

Reverse Proxy Solutions

There are many ways to use Traffic Server as a reverse proxy. Below are a few example scenarios. • Offload heavily-used origin servers. • Deliver content efficiently in geographically distant areas. • Provide security for origin servers that contain sensitive information.

Ofﬂoading Heavily-Used Origin Servers

Traffic Server can accept requests on behalf of the origin server and improve the speed and quality of web serving by reducing load and hot spots on backup origin servers. For example, a web hoster can maintain a scalable Traffic Server system with a set of low-cost, low-performance, less-reliable PC origin servers as backup servers. In fact, a single Traffic Server can act as the virtual origin server for multiple backup origin servers, as shown in the figure below.

Delivering Content in Geographically-Dispersed Areas

Traffic Server can be used in reverse proxy mode to accelerate origin servers that provide content to areas not located within close geographical proximity. Caches are typically easier to manage and are more cost-effective than replicating data. For example, Traffic Server can be used as a mirror site on the far side of a trans-Atlantic link to serve users without having to fetch the request and content across expensive, or higher latency, international connections. Unlike replication, for which hardware must be configured to replicate all data and to handle peak capacity, Traffic Server dynamically adjusts to optimally use the serving and storing capacity of the hardware. Traffic Server is also designed to keep content fresh automatically, thereby eliminating the complexity of updating remote origin servers.

3.3. 33 Apache-Trafﬁcserver-Server Documentation, latest

3.4: Trafﬁc Server as reverse proxy for a pair of origin servers

Providing Security for an Origin Server

Traffic Server can be used in reverse proxy mode to provide security for an origin server. If an origin server contains sensitive information that you want to keep secure inside your firewall, then you can use a Traffic Server outside the firewall as a reverse proxy for that origin server. When outside clients try to access the origin server, the requests instead go to Traffic Server. If the desired content is not sensitive, then it can be served from the cache. If the content is sensitive and not cacheable, then Traffic Server obtains the content from the origin server (the firewall allows only Traffic Server access to the origin server). The sensitive content resides on the origin server, safely inside the firewall.

How Does Reverse Proxy Work?

When a browser makes a request, it normally sends that request directly to the origin server. When Traffic Server is in reverse proxy mode, it intercepts the request before it reaches the origin server. Typically, this is done by setting up the DNS entry for the origin server (i.e., the origin server’s advertised hostname) so it resolves to the Traffic Server IP address. When Traffic Server is configured as the origin server, the browser connects to Traffic Server rather than the origin server. For additional information, see HTTP Reverse Proxy.

: To avoid a DNS conﬂict, the origin server’s hostname and its advertised hostname must not be the same.

HTTP Reverse Proxy

In reverse proxy mode, Traffic Server serves HTTP requests on behalf of a web server. The figure below illustrates how Traffic Server in reverse proxy mode serves an HTTP request from a client browser.

34 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

3.5: HTTP reverse proxy

The figure above demonstrates the following steps: 1. A client browser sends an HTTP request addressed to a host called www.host.com on port 80. Traffic Server receives the request because it is acting as the origin server (the origin server’s advertised hostname resolves to Traffic Server). 2. Traffic Server locates a map rule in the remap.config file and remaps the request to the specified origin server (realhost.com). 3. If the request cannot be served from cache, Traffic Server opens a connection to the origin server (or more likely, uses an existing connection it has pre-established), retrieves the content, and optionally caches it for future use. 4. If the request was a cache hit and the content is still fresh in the cache, or the content is now available through Traffic Server because of step 3, Traffic Server sends the requested object to the client from the cache directly.

: Trafﬁc Server, when updating its own cache from the origin server, will simultaneously deliver that content to the client while updating its cache database. The response to the client containing the requested object will begin as soon as Trafﬁc Server has received and processed the full response headers from the origin server.

To conﬁgure HTTP reverse proxy, you must perform the following tasks: • Create mapping rules in the remap.config ﬁle (refer to Creating Mapping Rules for HTTP Requests).

map http://www.host.com http://realhost.com

• Enable the reverse proxy option (refer to Enabling HTTP Reverse Proxy). In addition to the tasks above, you can also Setting Optional HTTP Reverse Proxy Options.

Handling Origin Server Redirect Responses

Origin servers often send redirect responses back to browsers redirecting them to different pages. For example, if an origin server is overloaded, then it might redirect browsers to a less loaded server. Origin servers also redirect when web pages have moved to different locations. When Traffic Server is configured as a reverse proxy, it must readdress redirects from origin servers so that browsers are redirected to Traffic Server and not to another origin server.

3.3. 35 Apache-Trafﬁcserver-Server Documentation, latest

To readdress redirects, Trafﬁc Server uses reverse-map rules. Unless you have :ts:cv:‘proxy.conﬁg.url_remap.pristine_host_hdr‘ enabled (the default) you should generally set up a reverse-map rule for each map rule. To create reverse-map rules, refer to Using Mapping Rules for HTTP Requests.

Using Mapping Rules for HTTP Requests

Trafﬁc Server uses two types of mapping rules for HTTP reverse proxy.

map rule

A map rule translates the URL in client requests into the URL where the content is located. When Traffic Server is in reverse proxy mode and receives an HTTP client request, it first constructs a complete request URL from the relative URL and its headers. Traffic Server then looks for a match by comparing the complete request URL with its list of target URLs in remap.config. For the request URL to match a target URL, the following conditions must be true: • The scheme of both URLs must be the same. • The host in both URLs must be the same. If the request URL contains an unqualified hostname, then it will never match a target URL with a fully-qualified hostname. • The ports in both URLs must be the same. If no port is specified in a URL, then the default port for the scheme of the URL is used. • The path portion of the target URL must match a prefix of the request URL path. If Traffic Server finds a match, then it translates the request URL into the replacement URL listed in the map rule: it sets the host and path of the request URL to match the replacement URL. If the URL contains path prefixes, then Traffic Server removes the prefix of the path that matches the target URL path and substitutes it with the path from the replacement URL. If two mappings match a request URL, then Traffic Server applies the first mapping listed in remap.config. reverse-map rule

A reverse-map rule translates the URL in origin server redirect responses to point to Traffic Server so that clients are redirected to Traffic Server instead of accessing an origin server directly. For example, if there is a directory /pub on an origin server at www.molasses.com and a client sends a request to that origin server for /pub, then the origin server might reply with a redirect by sending the Header Location: http://realhost.com/pub/ to let the client know that it was a directory it had requested, not a document (a common use of redirects is to normalize URLs so that clients can bookmark documents properly). Traffic Server uses reverse_map rules to prevent clients (that receive redirects from origin servers) from bypassing Traffic Server and directly accessing the origin servers. In many cases the client would be hitting a wall because realhost.com actually does not resolve for the client. (E.g.: Because it’s running on a port shielded by a firewall, or because it’s running on a non-routable LAN IP) Both map and reverse-map rules consist of a target (origin) URL and a replacement (destination) URL. In a map rule, the target URL points to Traffic Server and the replacement URL specifies where the original content is located. In a reverse-map rule, the target URL specifies where the original content is located and the replacement URL points to Traffic Server. Traffic Server stores mapping rules in remap.config located in the Traffic Server config directory.

Creating Mapping Rules for HTTP Requests

To create mapping rules:

36 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

1. Enter the map and reverse-map rules into remap.config. 2. Run the command traffic_ctl config reload to apply the conﬁguration changes.

Enabling HTTP Reverse Proxy

To enable HTTP reverse proxy: 1. Edit :ts:cv:‘proxy.conﬁg.reverse_proxy.enabled‘ in records.config.

CONFIG proxy.config.reverse_proxy.enabled INT1

2. Run the command traffic_ctl config reload to apply the conﬁguration changes.

Setting Optional HTTP Reverse Proxy Options

Traffic Server provides several reverse proxy configuration options in records.config that enable you to: • Configure Traffic Server to retain the client host header information in a request during translation. See :ts:cv:‘proxy.config.url_remap.pristine_host_hdr‘. • Configure Traffic Server to serve requests only to the origin servers listed in the mapping rules. As a result, requests to origin servers not listed in the mapping rules are not served. See :ts:cv:‘proxy.config.url_remap.remap_required‘. • Specify an alternate URL to which incoming requests from older clients ,such as ones that do not provide Host headers, are directed. See :ts:cv:‘proxy.config.header.parse.no_host_url_redirect‘. Run the command traffic_ctl config reload to apply any of these configuration changes.

Redirecting HTTP Requests

You can configure Traffic Server to redirect HTTP requests without having to contact any origin servers. For example, if you redirect all requests for http://www.ultraseek.com to http://www.server1.com/products/ portal/search/, then all HTTP requests for www.ultraseek.com go directly to www.server1.com/ products/portal/search. You can configure Traffic Server to perform permanent or temporary redirects. Permanent redirects notify the browser of the URL change (by returning the HTTP status code 301) so that the browser can update bookmarks. Temporary redirects notify the browser of the URL change for the current request only (by returning the HTTP status code 307 ). To set redirect rules: 1. For each redirect you want to set enter a mapping rule in remap.config. 2. Run the command traffic_ctl config reload to apply the configuration changes.

Example

The following permanently redirects all HTTP requests for www.server1.com to www.server2.com:

redirect http://www.server1.com http://www.server2.com

3.3. 37 Apache-Trafﬁcserver-Server Documentation, latest

Explicit Proxy Caching

If you want to use Traffic Server as an explicit proxy cache, you must configure client software (i.e., browsers) to send requests directly to Traffic Server. If you do not configure Traffic Server to use the transparency option (with which client requests are intercepted en route to origin servers by a switch/router and rerouted to the Traffic Server machine), then clients must configure their web browsers to send HTTP requests to the Traffic Server proxy cache by configuring their browsers to download proxy configuration instructions from a PAC file (Proxy Auto-Configuration file).

Conﬁguring Browsers Manually

To manually configure a browser to send HTTP requests to Traffic Server, clients must provide the following information: • The fully-qualified hostname or IP address of the Traffic Server node • The Traffic Server proxy server port (by default, 8080) In addition, clients can specify not to use Traffic Server for certain sites - in such cases, requests to the listed sites go directly to the origin server. The procedures for manual configuration vary among browser versions; refer to specific browser documentation for complete proxy configuration instructions. You do not need to set any special configuration options on Traffic Server if you want to accept requests from manually-configured browsers.

Using a PAC File

A PAC file is a specialized JavaScript function definition that a browser calls to determine how requests are handled. Clients must specify (in their browser settings) the URL from which the PAC file is loaded. You can store a PAC file on Traffic Server (or on any server in your network) and then provide the URL for this file to your clients.

Sample PAC File

The following sample PAC file instructs browsers to connect directly to all hosts without a fully-qualified domain name and to all hosts in the local domain. All other requests go to the Traffic Server named myproxy.company.com.:

function FindProxyForURL(url, host) { if (isPlainHostName(host))|| (localHostOrDomainIs(host,".company.com")) { return "DIRECT"; } else return "PROXY myproxy.company.com:8080; DIRECT"; }

Transparent Proxying

Building ATS for transparency

In most cases, if your environment supports transparency then configure will automatically enable it. For other environments you may need to twiddle the configure options.

38 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

--enable-posix-cap This enables POSIX capabilities, which are required for transparency. These are compiled in by default. To check your system, look for the header file sys/capability.h and the system library libcap. These are in the packages libcap and libcap-devel or libcap-dev (depending on the Distribution) contra-respectively. --enable-tproxy[=value] Enable TPROXY support, which is the Linux kernel feature used for transparency. This should be present in the base installation, there is no package associated with it. * auto Do automatic checks for the the TPROXY header file (linux/in.h) and enable TPROXY support if the IP_TRANSPARENT definition is present. This is the default if this option is not specified or value is omitted. * no Do not check for TPROXY support, disable support for it. * force Do not check for TPROXY support, enable it using the $ats@ built in value for IP_TRANSPARENT. This is useful for systems that have it in the kernel for but some reason do not have the appropriate system header file. * number Do not check for TPROXY support, use number as the IP_TRANSPARENT value. There are, at present, no known standard distributions of Linux that support TPROXY but use a value different from the built in ATS default. However, a custom built kernel may do so and in that case the specific value can be specified. In the default case, ATS configuration will automatically check for TPROXY support via the presence of the linux/ in.h header file and compile in TPROXY support if it is available. If that fails, you may be able to recover by using one of the options above. Note that transparency may be built in by default but it is not active unless explicitly enabled in the ATS configuration files.

Inline on a Linux Bridge

A Linux can be conﬁgured to operate in bridge mode. Two or more physical interfaces are assigned to the bridge. A single IP address is shared across the interfaces. By default any packet that arrives on one interface is immediately routed out another bridge interface. Linux packages required: • bridge-utils • ebtables In our example of setting up bridge mode we will use a local address of 192.168.1.11/24 and interfaces eth0 and eth1 as the bridge interfaces (more detailed documentation is available here). You may omit the ‘#’ character and everything after it. brctl addbr br0 # create bridge device brctl stp br0 off # Disable spanning tree protocol brctl addif br0 eth0 # Add eth0 to bridge brctl addif br0 eth1 # Add eth1 to bridge ifconfig eth00 0.0.0.0 # Get rid of interface IP addresses ifconfig eth10 0.0.0.0 # ditto # Set the bridge IP address and enable it ifconfig br0 192.168.1.11 netmask 255.255.255.0 up

If you have not already done so, remember to add a default route, such as this one for a gateway of 192.168.1.1. ip route add default via 192.168.1.1

At this point it is a good idea to test connectivity to verify the basic bridge is functional. Once the bridge is verified to work, this is the basic traffic pattern of interest. The green arrows are packets originating from the client and the red arrows are packets originating from the origin server. All traffic not directed to the local address will pass through the bridge. We need to break into some of the traffic and subject it to routing so that it can be routed to ATS. This requires ebtables. The flows we want to intercept are green 1 (from client to bridge) and red 1 (origin server to bridge).

3.3. 39 Apache-Trafﬁcserver-Server Documentation, latest

3.6: Picture of trafﬁc ﬂow through a bridge with ATS

In this example we will intercept port 80 (HTTP) traffic. We will use the BROUTING chain because it is traversed only for packets that originated externally and arrived on a (forwarding enabled) interface. Although it looks like this will intercept all port 80 traffic it will only affect the two flows described above. -j redirect marks the packet as being diverted to the bridge and not forwarded, and the DROP target puts the packets in the normal iptables routing so that we can use standard device tests on them1. Although this example handles only port 80, other ports are the same except for the port value. Note also the port here is the port from the point of view of the clients and origin servers, not the Traffic Server server port. ebtables-t broute-F # Flush the table # inbound traffic ebtables-t broute-A BROUTING-p IPv4--ip-proto tcp--ip-dport 80\ -j redirect--redirect-target DROP # returning outbound traffic ebtables-t broute-A BROUTING-p IPv4--ip-proto tcp--ip-sport 80\ -j redirect--redirect-target DROP

Trafﬁc Server operates at layer 3 so we need to use iptables to handle IP packets appropriately.: iptables-t mangle-A PREROUTING-i eth1-p tcp-m tcp--dport 80\ -j TPROXY--on-ip 0.0.0.0--on-port 8080--tproxy-mark1/1 iptables-t mangle-A PREROUTING-i eth0-p tcp-m tcp--sport 80\ -j MARK--set-mark1/1

At this point the directionality of the interfaces matters. For the example eth1 is the inbound (client side) interface, while eth0 is the outbound (origin server side) interface. We mark both flows of packets so that we can use policy routing on them. For inbound packets we need to use TPROXY to force acceptance of packets to foreign IP addresses. For returning outbound packets there will be a socket open bound to the foreign address, we need only force it to be delivered locally. The value for --on-ip is 0 because the target port is listening and not bound to a specific address. The value for --on-port must match the Traffic Server server port. Otherwise its value is arbitrary. --dport and --sport specify the port from the point of view of the clients and origin servers. Once the flows are marked we can force them to be delivered locally via the loopback interface via a policy routing table.:

1 The --redirect-target can be omitted, but then the iptables rules would need to use --physdev instead of just -i. The actual packet processing is identical.

40 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

ip rule add fwmark1/1 table1 ip route add local 0.0.0.0/0 dev lo table1

The marking used is arbitrary but it must be consistent between iptables and the routing rule. The table number must be in the range 1..253. To configure Traffic Server set the following values in records.config • :ts:cv:‘proxy.config.http.server_ports‘ value from --on-port (see below) • :ts:cv:‘proxy.config.reverse_proxy.enabled‘ 1 • :ts:cv:‘proxy.config.url_remap.remap_required‘ 0 You may also need to set :ts:cv:‘proxy.config.cluster.ethernet_interface‘ to “br0” (the name of the bridge interface from the Bridge Commands).

Additional troubleshooting

• Check to make sure that iptables is not ﬁltering (blocking) incoming HTTP connections. It is frequently the case that the default tables prevent incoming HTTP. You can clear all ﬁlters with the commands:

iptables-t filter--flush FORWARD iptables-t filter--flush INPUT

That is a bit drastic and should only be used for testing / debugging. A live system will likely need some filters in place but that is beyond the scope of this document. If this fixes the problem, then your filter set is too restrictive. Note that this problem will prevent the basic bridge (without ATS) from allowing HTTP traffic through. • Verify that IP packet forwarding is enabled. You can check this with:

cat/proc/sys/net/ipv4/ip_forward

The output should be a non-zero value (usually ‘1’). If it is zero, you can set it with:

echo'1'>/proc/sys/net/ipv4/ip_forward

This can setting can be persisted by putting it in /etc/sysctl.conf:

net/ipv4/ip_forward=1

Inline on a Linux router

The routed set up presumes the set of clients are on distinct networks behind a single physical interface. For the purposes of this example will we presume • The clients are on network 172.28.56.0/24 • The router connects the networks 172.28.56.0/24 and 192.168.1.0/24 • Interface eth0 is on the network 192.168.1.0/24

3.3. 41 Apache-Trafﬁcserver-Server Documentation, latest

• Interface eth1 is on the network 172.28.56.0/24 • The router is already configured to route traffic correctly for the clients. In this example we will intercept port 80 (HTTP) traffic that traverses the router. The first step is to use iptables to handle IP packets appropriately.

# reflow client web traffic to TPROXY iptables-t mangle-A PREROUTING-i eth1-p tcp-m tcp--dport 80-j TPROXY \ --on-ip 0.0.0.0--on-port 8080--tproxy-mark1/1 # Let locally directed traffic pass through. iptables-t mangle-A PREROUTING-i eth0--source 192.168.1.0/24-j ACCEPT iptables-t mangle-A PREROUTING-i eth0--destination 192.168.1.0/24-j ACCEPT # Mark presumed return web traffic iptables-t mangle-A PREROUTING-i eth0-p tcp-m tcp--sport 80-j MARK--set-mark

˓→1/1

We mark packets so that we can use policy routing on them. For inbound packets we use TPROXY to make it possible to accept packets sent to foreign IP addresses. For returning outbound packets there will be a socket open bound to the foreign address, we need only force it to be delivered locally. The value for --on-ip is 0 because the target port is listening and not bound to a specific address. The value for --on-port must match the Traffic Server server port. Otherwise its value is arbitrary. --dport and --sport specify the port from the point of view of the clients and origin servers. The middle two lines exempt local web traffic from being marked for Traffic Server – these rules can be tightened or loosened as needed. They server by matching traffic and exiting the iptables processing via ACCEPT before the last line is checked. Once the flows are marked we can force them to be delivered locally via the loopback interface via a policy routing table.

ip rule add fwmark1/1 table1 ip route add local 0.0.0.0/0 dev lo table1

WCCP Conﬁguration

WCCP is de-facto semi-standard used by routers to redirect network traffic to caches. It is available on most Cisco™ routers although it does not appear to be officially supported by Cisco. The primary benefits of WCCP are • If already have a router that supports WCCP inline you do not have to change your network topology. • WCCP fails open so that if the Traffic Server machine fails, it is bypassed and users continue to have Internet access. Use of WCCP only makes sense for client side transparency1 because if the clients are explicitly proxied by Traffic Server there’s no benefit to WCCP fail open, as the clients will continue to directly access the unresponsive Traffic Server host. It would be better to adjust the routing tables on the router for explicit proxying.

1 Server side transparency should also be used, but it is not as significant. In its absence, however, origin servers may see the source address of connections suddenly change from the Traffic Server address to client addresses, which could cause problems. Further, the primary reason for not having server side transparency is to hide client addresses which is defeated if the Traffic Server host fails.

42 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Because the router serves as the inline network element, Traffic Server must run on a separate host. This host can be located anywhere as long as Traffic Server is either on the same network segment or a GRE tunnel can be maintained between the Traffic Server host and the router.

This document presumes that the router is already properly configured to handle traffic between the clients and the origin servers. If you are not certain, verify it before attempting to configure Traffic Server with WCCP. This is also a good state to which to revert should the configuration go badly.

Conﬁguration overview

Setting WCCP is a three step process, first configuring the router, the Traffic Server host, and Traffic Server.

The router will not respond to WCCP protocol packets unless explicitly configured to do so. Via WCCP, the router can be made to perform packet interception and redirection needed by Traffic Server transparency. The WCCP protocol in effect acts as means of controlling a rough form of policy routing with positive heartbeat cutoff. The Traffic Server host system must also be configured using iptables to accept connections on foreign addresses. This is done roughly the same way as the standard transparency use. Finally Traffic Server itself must be configured for transparency and use of WCCP. The former is again very similar to the standard use, while WCCP configuration is specific to WCCP and uses a separate configuration file, referred to by the records.config file. The primary concern for configuration in which of three basic topologies are to be used. • Dedicated – Traffic Server traffic goes over an interface that is not used for client nor origin server traffic. • Shared – Traffic Server traffic shares an interface with client or server traffic. • Inside Shared – Traffic Server and client traffic share an interface. • Outside Shared – Traffic Server and origin server traffic share an interface. In general the dedicated topology is preferred. However, if the router has only two interfaces one of the shared topologies will be required2 Click the links above for more detailed configuration information on a specific topology.

Shared interface issues

A shared interface topology has additional issues compared to a dedicated topology that must be handled. Such a topology is required if the router has only two interfaces, and because of these additional issues it is normally only used in such cases, although nothing prevents it use even if the router has three or more interfaces. The basic concept for a shared interface is to use a tunnel to simulate the dedicated interface case. This enables the packets to be distinguished at layer 3. For this reason, layer 2 redirection cannot be used because the WCCP configuration cannot distinguish between packets returning from the origin server and packets returning from Traffic Server as they are distinguished only by layer 2 addressing3. Fortunately the GRE tunnel used for packet forwarding and return can be used as the simulated interface for Traffic Server.

Frequently encountered problems

2 If your router has only one interface, it’s hardly a router. 3 This is not fundamentally impossible, as the packets are distinct in layer

3.3. 43 Apache-Trafﬁcserver-Server Documentation, latest

MTU and fragmentation

In most cases the basic conﬁguration using a tunnel in any topology can fail due to issues with fragmentation. The socket logic is unable to know that its packets will eventually be put in to a tunnel which will by its nature have a smaller MTU than the physical interface which it uses. This can lead to pathological behavior or outright failure as the packets sent are just a little too big. It is not possible to solve easily by changing the MTU on the physical interface because the tunnel interface uses that to compute its own MTU.

References

• WCCP Router Conﬁguration Commands - IOS 12.2

WCCP Service Conﬁguration

The service definition file is used by traffic_wccp and traffic_server directly. The elements in the security definition file are inspired by the WCCP RFC (8/2012). There is also an older version of the RFC that shows up commonly in search results, WCCP (4/2001), and was the RFC reference used in the original WCCP support for Traffic Server several years ago. A sample service group file is included in the source tree under traffic_wccp.

Security Section

In the security section, you can define a password shared between the WCCP Client and the WCCP router. This password is used to encrypt the WCCP traffic. It is optional, but highly recommended. Attributes in this section • option - Must be set to MD5 if you want to encrypt your WCCP traffic • key – The same password that you set with the associated WCCP router.

Services Section

In the services section you can define a list of service groups. Each top level entry is a separate service group. Service group attributes include • name – A name for the service. Not used in the rest of the WCCP processing. • description – A description of the service. Again, not used in the rest of the WCCP processing. • id - The security group ID. It must match the service group ID that has been defined on the associated WCCP router. This is the true service group identifier from the WCCP perspective. • type – This defines the type of service group either “STANDARD” or “DYNAMIC”. There is one standard defined service group, HTTP with the id of 0. The 4/2001 RFC indicates that id’s 0-50 are reserved for well known service groups. But more recent 8/2012 RFC indicates that values 0 through 254 are valid service id’s for dynamic services. To avoid differences with older WCCP routers, you probably want to avoid dynamic service ID’s 0 through 50. • priority – This is a value from 0 to 255. The higher number is a higher priority. Well known (STANDARD) services are set to a value of 240. If there are multiple service groups that could match a given packet, the higher priority service group is applied. RFC For example, you have service group 100 defined for packets

44 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

with destination port 80, and service group 101 defined for packets with source port 1024. For a packet with destination port set to 80 and source port set to 1024, the priorities of the service groups would need to be compared to determine which service group applies. • protocol – This is IP protocol number that should match. Generally this is set to 6 (TCP) or 17 (UDP). • assignment – WCCP supports multiple WCCP clients supporting a single service group. However, the current WCCP client implementation in Traffic Server assumes there is only a single WCCP client, and so creates assignment tables that will direct all traffic to that WCCP client. The assignment type is either hash or mask, and if it is not set, it defaults to hash. If Traffic Server ever supports more than one cache, it will likely only support a balanced hash assignment. The mask/value assignment seems to be better suited to situations where the traffic needs to be more strongly controlled. • primary-hash – This is the element of the packet that is used to compute the primary key. The value options are src_ip, dst_ip, src_port, or dst_port. This entry is a list, so multiple values can be specified. In that case, all the specified packet attributes will be used to compute the hash bucket. In the current implementation, the primary hash value does not matter, since the client always generates a hash table that directs all matching traffic to it. But if multiple clients are ever supported, knowledge of the local traffic distribution could be used to pick a packet attribute that will better spread traffic over the WCCP clients. • alt-hash – The protocol supports a two level hash. This attribute is a list with the same value options as for primary-hash. Again, since the current Traffic Server implementation only creates assignment tables to a single client, specifying the alt-hash values does nothing. • ports – This is a list of port values. Up to 8 port values may be included in a service group definition. • port-type – This attribute can have the value of src or dst. If not specified, it defaults to dst. It indicates whether the port values should be interpreted as source ports or destination ports. • forward – This is a list. The list of the values can be GRE or L2. This advertises how the client wants to process WCCP packets. GRE means that the packets will be delivered in a GRE tunnel. This is the default. L2 means that the client is on the same network and can get traffic delivered to it from the router by L2 routing (MAC addresses). • return – The WCCP protocol allows a WCCP client to decline a packet and return it back to the router. The current WCCP client implementation never does this. The value options are the same as for the forward attribute. • routers – This is the list of router addresses the WCCP client communicates with. The WCCP protocols allows for multiple WCCP routers to be involved in a service group. The multiple router scenario has at most been lightly tested in the Traffic Server implementation. • proc-name – This attribute is only used by traffic_wccp. It is not used in the traffic_server WCCP support. This is the path to a process’ PID file. The service group is advertised to the WCCP router if the process identified in the PID file is currently operational. Transparent Proxying is the ability of a proxy (such as ATS) to intercept connections between clients and servers without being visible. The general network structure that will be used in this documentation is shown in the following figure. There must be a gateway device through which all network traffic passes from the client to the Internet (or external cloud). The gateway is responsible for effectively splicing ATS in to selected streams of that traffic. Each traffic stream is split in two, with ATS terminating both sides. That is, stream green-1, red-2, is split in to the green connection and the red connection. Note that ATS may or may not be on the gateway system, the redirected traffic can flow over other network infrastructure. Because ATS uses two connections, transparency can be set independently on the client and origin server (Internet / external cloud) side. We will define what is generally called “transparent proxy” as two aspects, inbound transparency and outbound transparency.

3.3. 45 Apache-Trafﬁcserver-Server Documentation, latest

3.7: ATS basic trafﬁc ﬂow of Transparent Proxy

Inbound transparency is a proxy that is transparent to connections that are inbound to the proxy, i.e. a connection initiated by a client which connects to the proxy (green-1). Similarly, outbound transparency is a proxy that is transparent to connections that are outbound from the proxy, i.e. a connection initiated by the proxy to an origin server (red-2). In most situations these two types of transparency are combined, but that is not required. Trafﬁc Server supports transparency independently on the two sides.

: It is critical to note that any transparency requires specialized routing and cannot be done solely by conﬁguring ATS. ATS transparency also requires support from the Linux kernel and therefore currently only works on sufﬁciently recent Linux kernels that support the following features: • TPROXY • POSIX capabilities In addition the specialized routing will require using iptables and in some cases ebtables.

Standard build procedures should work for transparency support but if not consult these more detailed instructions. Transparency is configured per server port, not globally. This is done via the configuration values :ts:cv:‘proxy.config.http.server_ports‘. In addition, :ts:cv:‘proxy.config.reverse_proxy.enabled‘ must be enabled if the client side is transparent. That should be fixed in a future patch. In the first case use the attribute character (replacing the default ‘X’) Attribute Transparency Style Reverse Proxy = Full transparency: either > Inbound (client) transparency: enabled < Outbound (origin server) transparency: either In the outbound transparent case clients must connect directly to ATS either through an explicit proxy mechanism or by advertising the IP address of the ATS server via DNS as the origin server address. Some tested scenarios – • Inline on a Linux Bridge • Inline on a Linux router

46 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

• WCCP Conﬁguration

Forward Proxy

The Apache Traffic Server is a general purpose proxy, configurable as both a reverse and forward proxy. A forward proxy can be used as a central tool in your infrastructure to access the web and it may be combined with a cache to lower your overall bandwidth usage. Forward proxies act as a gatekeeper between client browsers on your local network and all (or some, at your configuration’s discretion) web sites accessed by those clients. The forward proxy will receive the HTTP requests, perform any filtering or request alteration rules you establish, and when appropriate forward the request on to its destination website. The response will return through your proxy, where it may optionally be cached and/or modified, and then returned to the original client. There are two modes in which your forward proxy may operate: Forward Proxy Each client must be configured explicitly to use the forward proxy. Client browsers will be aware of the fact they are using a proxy and will form their HTTP requests appropriately. This results in the initial HTTP command being issued with fully qualified URIs that contain the destination hostname:

GET http://example.com/index.php?id=123 HTTP/1.1

Transparent Proxy The use of a transparent proxy is typically done in concert with network routing rules which redirect all outbound HTTP trafﬁc through your proxy. Clients will behave, and form their HTTP requests, as if they are contacting the remote site directly, and will not be aware of the existence of a proxy server in between themselves and the remote servers. HTTP requests will be generated per their usual form, with only paths in the command and a separate Host request header:

GET /index?id=123 HTTP/1.1 Host: example.com

Apache Trafﬁc Server may be conﬁgured to operate as both a forward and a transparent proxy simultaneously.

Proxy Conﬁguration

Configuring basic forward proxy operation in Traffic Server is quite simple and straightforward. 1. Permit Traffic Server to process requests for hosts not explicitly configured in the remap rules, by modifying :ts:cv:‘proxy.config.url_remap.remap_required‘ in records.config:

CONFIG proxy.config.url_remap.remap_required INT0

2. Optional: If Trafﬁc Server will be operating strictly as a forward proxy, you will want to disable reverse proxy support by modifying :ts:cv:‘proxy.conﬁg.reverse_proxy.enabled‘ in records.config:

CONFIG proxy.config.reverse_proxy.enabled INT0

You may also want to consider some of these configuration options: • Setting :ts:cv:‘proxy.config.http.no_dns_just_forward_to_parent‘ determines which host will be used for DNS resolution. • Proxy Authentication can be enabled or disabled with :ts:cv:‘proxy.config.http.forward.proxy_auth_to_parent‘ should you also be employing a proxy cache. • The client request header X-Forwarded-For may be toggled with :ts:cv:‘proxy.config.http.insert_squid_x_forwarded_for‘.

3.3. 47 Apache-Trafﬁcserver-Server Documentation, latest

Client Conﬁguration

If you are operating your proxy in transparent mode, your clients should require no special proxy-related configuration. If you are operating in explicit forward proxy mode, without automatic routing rules on your network to direct all outbound traffic through the proxy, your client browsers will need to be directed to the proxy. This may be accomplished in two different ways. Clients may be configured to use the default 8080 port on your Traffic Server host as a proxy. This will result in all requests from that client browser being issued through the single forward proxy as configured.

Security Considerations

It’s important to note that once your Apache Traffic Server is configured as a forward proxy it will indiscriminately accept proxy requests from anyone. If it is reachable from the Internet, then you have configured an Open Proxy. This is generally not desirable, as it will permit anyone to potentially use your network as the source of traffic to sites of their choosing. To avoid this, you’ll have to make sure your proxy server is either only reachable from within your private network or is secured by firewall rules that permit only those you wish to have access to the proxy.

Trafﬁc Server Cluster

Trafﬁc Server scales from a single node to multiple nodes that form a cluster allowing you to improve system performance and reliability.

Understanding Trafﬁc Server Clusters

A Traffic Server cluster consists of multiple Traffic Server nodes. The nodes in a cluster share configuration information and can form a single logical cache. Traffic Server detects the addition and deletion of nodes in the cluster automatically and can detect when a node is unavailable. Traffic Server uses its own protocol for clustering, which is multicast for node location and heartbeat, but unicast for all data exchange within the cluster. Traffic Server has two clustering modes: • Management-only mode; refer to Management-Only Clustering below. • Full-clustering mode; refer to Full Clustering

Management-Only Clustering

In management-only clustering mode, Traffic Server cluster nodes share configuration information. You can administer all the nodes at the same time. Traffic Server uses a multicast management protocol to provide a single system image of your Traffic Server cluster. Information about cluster membership, configuration, and exceptions is shared across all nodes, and the traffic_manager process automatically propagates configuration changes to all the nodes.

Full Clustering

In full-clustering mode, as well as sharing configuration information, a Traffic Server cluster distributes its cache across its nodes into a single, virtual object store, rather than replicating the cache node by node. Traffic Server can provide an enormous aggregate cache size and can maximize cache hit rate by storing objects only once across the entire cluster. A fully clustered Traffic Server maps objects to specific nodes in the cluster. When a node receives a request, it checks to see if the request is a hit somewhere in the cluster. If the request is a hit on a different node, the node handling

48 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

the request obtains the object from the hit node and serves it to the client. Trafﬁc Server uses its own communication protocol to obtain an object from sibling cluster nodes. If a node fails or is shut down and removed, Trafﬁc Server removes references to the missing node on all nodes in the cluster. Full clustering recommends a dedicated network interface for cluster communication to achieve better performance.

Enabling Clustering Mode

Before a node is added into a cluster, please ensure that the following conditions are being met: • All nodes are running the same operating system: – The same distribution, e.g.: RHEL 6.6 – The same kernel, e.g.: 2.6.32-504.23.4.el6 – The same architecture, e.g.: x86_64 • All nodes have the same version of Traffic Server installed • All nodes are composed of the same hardware • All nodes are on the same switch or same VLAN. Traffic Server does not apply the clustering mode change to all the nodes in the cluster. You must change the clustering mode on each node individually. You may following these instructions: 1. Setup the same cluster name, with :ts:cv:‘proxy.config.proxy_name‘, e.g. MyCluster. 2. Set :ts:cv:‘proxy.local.cluster.type‘ to 1 to enable cluster mode. The following values of this configuration are valid: Value Description 1 Full-Clustering mode 2 Management-Only mode 3 No clustering (default) 3. Configure :ts:cv:‘proxy.config.cluster.ethernet_interface‘, e.g.: eth0. This should be replaced with the node’s real interface. We recommends a dedicated physical interface here. Refer to :ts:cv:‘proxy.local.cluster.type‘ for a full description. 4. Enable configuration changes:

traffic_ctl config reload

5. Restart trafﬁc server:

traffic_ctl server restart

The traffic_server and traffic_manager processes will need to restart after the change of :ts:cv:‘proxy.local.cluster.type‘ and :ts:cv:‘proxy.conﬁg.cluster.ethernet_interface‘ have taken place. Trafﬁc Server will join the cluster in about 10 seconds. To verify the hosts in the cluster, you can run:

traffic_ctl metric get proxy.process.cluster.nodes

Cluster node status is also tracked in cluster.config in the conﬁguration directory. This conﬁguration is generated by the system and should not be edited. It contains a list of the machines that are currently members of the cluster, for example:

3.3. 49 Apache-Trafﬁcserver-Server Documentation, latest

# 3 127.1.2.3:80 127.1.2.4:80 127.1.2.5:80

After successfully joining a cluster, all changes of global conﬁgurations performed on any node in that cluster will take effect on all nodes, removing the need to manually duplicate conﬁguration changes across each node individually.

Deleting Nodes from a Cluster

To delete a node from the Trafﬁc Server cluster, return the setting :ts:cv:‘proxy.local.cluster.type‘ to the default value 3 and reload.

Common issues for Cluster setup

1. The Cluster member auto discovery is built upon multi-casting UDP, and as such is impossible to setup where multi-casting is not avaliable, such as AWS EC2. 2. The Cluster will depend on ports 8088, 8089, and 8086. These ports must not be blocked by any network conﬁgurations or ﬁrewalls on the network used for internal cluster communication.

Performance Tuning for Busy Clusters

Beginning with version 3.2.0, Apache Trafﬁc Server can handle multiple internal cluster connections and the number of Cluster Threads is conﬁgurable. Each of the threads will keep one connection open to all peering cluster nodes.

Increasing Cluster Threads

In the cluster environment, the current performance of the cluster threads will consume the same cpu usage as a normal network thread. It’s reasonable to keep roughly the same number of cluster threads as network threads. For example, if you are running a system with 10 network processing threads, you can set the number of cluster threads by modifying :ts:cv:‘proxy.conﬁg.cluster.threads‘ to 10: traffic_ctl config set proxy.config.cluster.threads 10

Hierarchical Caching

Understanding Cache Hierarchies

A cache hierarchy consists of cache levels that communicate with each other. Traffic Server supports several types of cache hierarchies. All cache hierarchies recognize the concept of parent and child. A parent cache is a cache higher up in the hierarchy, to which Traffic Server can forward requests. A child cache is a cache for which Traffic Server is a parent. Traffic Server supports the following hierarchical caching options:

50 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Parent Caching

If a Traffic Server node cannot find a requested object in its cache, then it searches a parent cache (which itself can search other caches) before finally retrieving the object from the origin server. You can configure a Traffic Server node to use multiple parent caches so that if one parent is unavailable, the other parent caches will be checked in turn until either the request is serviced properly or no further parent caches are available and the origin server is contacted. This is called Parent Failover. Traffic Server supports parent caching for both HTTP and HTTPS requests. If you do not want all requests to go to the parent cache, then simply configure Traffic Server to route certain requests (such as requests containing specific URLs) directly to the origin server. This may be achieved by setting parent proxy rules in parent.config. The figure below illustrates a simple cache hierarchy with a Traffic Server node configured to use a parent cache. In the following scenario, a client sends a request to a Traffic Server node that is a child in the cache hierarchy (because it’s configured to forward missed requests to a parent cache). The request is a cache miss, so Traffic Server then forwards the request to the parent cache where it is a cache hit. The parent sends a copy of the content to the Traffic Server, where it is cached and then served to the client. Future requests for this content can now be served directly from the Traffic Server cache (until the data is stale or expired).

3.8: Parent caching

If the request is a cache miss on the parent, then the parent retrieves the content from the origin server (or from another cache, depending on the parent’s conﬁguration). The parent caches the content and then sends a copy to Trafﬁc Server (its child), where it is cached and served to the client.

3.3. 51 Apache-Trafﬁcserver-Server Documentation, latest

Interaction with Remap.conﬁg

If remap rules are required (:ts:cv:‘proxy.config.reverse_proxy.enabled‘), when a request comes in to a child node, its remap.config is evaluated before parent selection. This means that the client request is translated according to the remap rule, and therefore, any parent selection should be made against the remapped host name. This is true regardless of pristine host headers (:ts:cv:‘proxy.config.url_remap.pristine_host_hdr‘) being enabled or not. The parent node will receive the translated request (and thus needs to be configured to accept it).

Example

The client makes a request to Trafﬁc Server for http://example.com. The origin server for the request is http://origin. example.com; the parent node is parent1.example.com, and the child node is conﬁgured as a reverse proxy. If the child’s remap.config contains map http://example.com http://origin.example.com with the child’s parent.config containing dest_domain=origin.example.com method=get parent="parent1.example.com:80 ) and parent cache (parent1.example.com) would need to have a remap.config line similar to map http://origin.example.com http://origin.example.com With this example, if parent1.example.com is down, the child node would automatically directly contact the origin. example.com on a cache miss.

Parent Failover

Traffic Server supports use of several parent caches. This ensures that if one parent cache is not available, another parent cache can service client requests. When you configure your Traffic Server to use more than one parent cache, Traffic Server detects when a parent is not available and sends missed requests to another parent cache. If you specify more than two parent caches, then the order in which the parent caches are queried depends upon the parent proxy rules configured in the parent.config configuration file. By default, the parent caches are queried in the order they are listed in the configuration file.

Conﬁguring Trafﬁc Server to Use a Parent Cache

To configure Traffic Server to use one or more parent caches, you must perform the configuration adjustments detailed below.

: You need to configure the child cache only. Assuming the parent nodes are configured to serve the child’s origin server, no additional configuration is needed for the nodes acting as Traffic Server parent caches.

1. Enable the parent caching option by adjusting :ts:cv:‘proxy.conﬁg.http.parent_proxy_routing_enable‘ in records.config.

CONFIG proxy.config.http.parent_proxy_routing_enable INT1

2. Identify the parent cache you want to use to service missed requests. To use parent failover, you must identify more than one parent cache so that when a parent cache is unavailable, requests are sent to another parent cache.

52 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

3. Edit parent.config to set parent proxy rules which will specify the parent cache to which you want missed requests to be forwarded. The following example conﬁgures Trafﬁc Server to route all requests containing the regular expression politics and the path /viewpoint directly to the origin server (bypassing any parent hierarchies):

url_regex=politics prefix=/viewpoint go_direct=true

The following example configures Traffic Server to direct all missed requests with URLs beginning with http:/ /host1 to the parent cache parent1. If parent1 cannot serve the requests, then requests are forwarded to parent2. Because round-robin=true, Traffic Server goes through the parent cache list in a round-robin based on client IP address.:

dest_host=host1 scheme=http parent="parent1;parent2" round-robin=strict

Run the command traffic_ctl config reload to apply the conﬁguration changes.

ICP Peering

The Internet Cache Protocol (ICP) is used by proxy caches to exchange information about their content. ICP query messages ask other caches if they are storing a particular URL; ICP response messages reply with a hit or miss answer. A cache exchanges ICP messages only with specific ICP peers, which are neighboring caches that can receive ICP messages. An ICP peer can be a sibling cache (which is at the same level in the hierarchy) or a parent cache (which is one level up in the hierarchy). If Traffic Server has ICP caching enabled, then it sends ICP queries to its ICP peers when the HTTP request is a cache miss. If there are no hits but parents exist, then a parent is selected using a round-robin policy. If no ICP parents exist, then Traffic Server forwards the request to its HTTP parents. If there are no HTTP parent caches established, then Traffic Server forwards the request to the origin server. If Traffic Server receives a hit message from an ICP peer, then Traffic Server sends the HTTP request to that peer. However, it might turn out to be a cache miss because the original HTTP request contains header information that is not communicated by the ICP query. For example, the hit might not be the requested alternate. If an ICP hit turns out to be a miss, then Traffic Server forwards the request to either its HTTP parent caches or to the origin server. To configure a Traffic Server node to be part of an ICP cache hierarchy, you must perform the following tasks: • Determine if the Traffic Server can receive ICP messages only, or if it can send and receive ICP messages. • Determine if Traffic Server can send messages directly to each ICP peer or send a single message on a specified multicast channel. • Specify the port used for ICP messages. • Set the ICP query timeout. • Identify the ICP peers (siblings and parents) with which Traffic Server can communicate. To configure Traffic Server to use an ICP cache hierarchy edit the following variables in records.config file: • :ts:cv:‘proxy.config.icp.enabled‘ • :ts:cv:‘proxy.config.icp.icp_port‘ • :ts:cv:‘proxy.config.icp.multicast_enabled‘ • :ts:cv:‘proxy.config.icp.query_timeout‘ Edit icp.config file located in the Traffic Server config directory: For each ICP peer you want to identify, enter a separate rule in the icp.config file. Run the command traffic_ctl config reload to apply the configuration changes.

3.3. 53 Apache-Trafﬁcserver-Server Documentation, latest

Interacting with Trafﬁc Server

Trafﬁc Line

Stats Over HTTP

Cache Inspector

Security

Controlling Access

Traffic Server can be configured to allow only certain clients to use the proxy cache. 1. Add a line to ip_allow.config for each IP address or range of IP addresses allowed to access Traffic Server. 2. Run the command traffic_ctl config reload to apply the configuration changes.

SSL Termination

The Traffic Server SSL termination option enables you to secure connections in reverse proxy mode between a client and a Traffic Server and/or Traffic Server and an origin server. The following sections describe how to enable and configure the SSL termination option. • Enable and configure SSL termination for client/Traffic Server connections: Client and Traffic Server Connec- tions • Enable and configure SSL termination for Traffic Server/origin server connections: Traffic Server and Origin Server Connections

Client and Trafﬁc Server Connections

The figure below illustrates communication between a client and Traffic Server (and between Traffic Server and an origin server) when the SSL termination option is enabled and configured for Client/Traffic Server connections only. The figure above depicts the following: 1. The client sends an HTTPS request for content. Traffic Server receives the request and performs the SSL handshake to authenticate the client (depending on the authentication options configured) and determine the encryption method that will be used. If the client is allowed access, then Traffic Server checks its cache for the requested content. 2. If the request is a cache hit and the content is fresh, then Traffic Server encrypts the content and sends it to the client. The client decrypts the content (using the method determined during the handshake) and displays it. 3. If the request is a cache miss or cached content is stale, then Traffic Server communicates with the origin server via HTTP and obtains a plain text version of the content. Traffic Server saves the plain text version of the content in its cache, encrypts the content, and sends it to the client. The client decrypts and displays the content. To configure Traffic Server to use the SSL termination option for Client/Traffic Server connections, you must do the following: 1. Obtain and install an SSL server certificate from a recognized certificate authority. The SSL server certificate contains information that enables the client to authenticate Traffic Server and exchange encryption keys.

54 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

3.9: Client and Trafﬁc Server communication using SSL termination

2. Set the port number used for SSL communication using :ts:cv:‘proxy.conﬁg.http.server_ports‘ in records. config. 3. Set the appropriate base path for your SSL certiﬁcates and private keys in records.config.

CONFIG proxy.config.ssl.server.cert.path STRING"/opt/ts/etc/ssl/certs/" CONFIG proxy.config.ssl.server.private_key.path STRING"/opt/ts/etc/ssl/keys/"

4. Add an entry to ssl_multicert.config for each certiﬁcate and key which your Trafﬁc Server system will be using to terminate SSL connections with clients.

ip_dest=1.2.3.4 ssl_cert_name=example.com.pem ip_dest=* ssl_cert_name=default.pem

5. Optional: Configure the use of client certificates using the variable :ts:cv:‘proxy.config.ssl.client.certification_level‘ in records.config. If you configure Traffic Server to require client certificates, then Traffic Server verifies the client certificate during the SSL handshake that authenticates the client. If you configure Traffic Server to not require client certificates, or if you configure certificates to be optional and the connecting client does not present one, then access to Traffic Server is managed through other Traffic Server options that have been set (such as rules in ip_allow.config).

CONFIG proxy.config.ssl.client.certification_level INT0

This variable permits one of the following values to be set: Value Description 0 Client certificates not required. 1 Client certificates optional. If present, will be used to validate. 2 Client certficates required, and must validate based on configured CAs. 6. Optional: Configure the use of Certification Authorities (CAs). CAs add security by verifying the identity of the person requesting a certificate. The list of acceptable CA signers is configured with :ts:cv:‘proxy.config.ssl.CA.cert.path‘ in records.config.

CONFIG proxy.config.ssl.CA.cert.path STRING"/opt/CA/certs/private-ca.pem"

7. Run the command traffic_ctl server restart to restart Trafﬁc Server on the local node or traffic_ctl cluster restart to restart Trafﬁc Server on all the nodes in a cluster.

3.5. Security 55 Apache-Trafﬁcserver-Server Documentation, latest

Trafﬁc Server and Origin Server Connections

The figure below illustrates communication between Traffic Server and an origin server when the SSL termination option is enabled for Traffic Server/origin server connections.

3.10: Trafﬁc Server and origin server communication using SSL termination

The figure above depicts the following: 1. If a client request is a cache miss or is stale, then Traffic Server sends an HTTPS request for the content to the origin server. The origin server receives the request and performs the SSL handshake to authenticate Traffic Server and determine the encryption method to be used. 2. If Traffic Server is allowed access, then the origin server encrypts the content and sends it to Traffic Server, where it is decrypted (using the method determined during the handshake). A plain text version of the content is saved in the cache, if Traffic Server deems the content cacheable. 3. If SSL termination is enabled for Client/Traffic Server connections, then Traffic Server re-encrypts the content and sends it to the client via HTTPS, where it is decrypted and displayed. If SSL termination is not enabled for Client/Traffic Server connections, then Traffic Server sends the plain text version of the content to the client via HTTP. To configure Traffic Server to use the SSL termination option for Traffic Server and origin server connections, you must do the following: 1. Ensure first that your origin server responds properly to SSL requests, and configure it for client certificate validation if you intend to use that as part of your access control scheme. Refer to your origin server’s documentation for details. If your origin server is another Traffic Server system, then you may follow the steps outlined in Client and Traffic Server Connections for configuring the origin server to validate client certificates. 2. Optional: Obtain and install an SSL client certificate from a recognized certificate authority, if your origin server requires client certificate validation for access control. Your client certificate must be signed by a Certificate Authority recognized by your origin server. If you are using a client certificate, you must add its location to records.config in the setting :ts:cv:‘proxy.config.ssl.client.cert.path‘ and :ts:cv:‘proxy.config.ssl.client.cert.filename‘.

CONFIG proxy.config.ssl.client.cert.path STRING"/opt/ts/etc/ssl/certs/" CONFIG proxy.config.ssl.client.cert.filename STRING"client.pem"

You must also provide the paths to the private key for this certificate, unless the key is contained within the same file as the certificate, using :ts:cv:‘proxy.config.ssl.client.private_key.path‘ and :ts:cv:‘proxy.config.ssl.client.private_key.filename‘.

56 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

CONFIG proxy.config.ssl.client.private_key.path STRING"/opt/ts/etc/ssl/keys/" CONFIG proxy.config.ssl.client.private_key.filename STRING"client.pem"

3. Enable or disable, per your security policy, server SSL certificate verification using :ts:cv:‘proxy.config.ssl.client.verify.server‘ in records.config.

CONFIG proxy.config.ssl.client.verify.server INT1

4. Add the collection of authorized Certificate Authorities to the Traffic Server configuration in records.config using the settings :ts:cv:‘proxy.config.ssl.client.CA.cert.path‘ and :ts:cv:‘proxy.config.ssl.client.CA.cert.filename‘.

CONFIG proxy.config.ssl.client.CA.cert.path STRING"/opt/ts/etc/ssl/certs/" CONFIG proxy.config.ssl.client.CA.cert.filename STRING"CAs.pem"

5. Run the command traffic_ctl server restart to restart Trafﬁc Server on the local node or traffic_ctl cluster restart to restart Trafﬁc Server on all the nodes in a cluster.

Rotating TLS Session Ticket Keys

TLS sessions can be resumed through session tickets which are encrypted with a session ticket key and stored on clients. For better security, the ticket keys can be rotated periodically, say, every 24 hours. The ticket keys are stored in a ticket key file as a reverse queue in 48-byte chunks. 1. Generate a new ticket key and push it to the beginning of the ticket key file. 2. Optional: Delete the last ticket key from the ticket key file. 3. Touch ssl_multicert.config to indicate that the SSL configuration is stale. 4. Run the command traffic_ctl config reload to apply the new ticket key.

OCSP Stapling

OCSP Stapling is an alternative approach to checking the revocation status of an SSL certificate using the Online Certificate Status Protocol. Under the original OCSP implementation, clients requested a certificate’s revocation status directly from the Certificate Authority (CA) that issued the certificate. This could cause significant load on the CA servers since they were required to provide a response to every client of a given certificate in real time. Enabling OCSP Stapling instructs Traffic Server to retrieve and cache the revocation status of all configured SSL certificates, and present them to the client when the client requests the status. Traffic Server will automatically query the OCSP responder specified in the SSL certificate to gather the latest revocation status. Traffic Server will then cache the results for each configured certifcate. The location of the OCSP responder is taken from the Authority Information Access field of the signed certificate. For example:

Authority Information Access: OCSP- URI:http://ocsp.digicert.com CA Issuers- URI:http://cacerts.digicert.com/DigiCertSHA2SecureServerCA.

˓→crt

Support for OCSP Stapling can be tested using the -status option of the OpenSSL client:

3.5. Security 57 Apache-Trafﬁcserver-Server Documentation, latest

$ openssl s_client -connect mozillalabs.com:443 -status ... ======OCSP Response Data: OCSP Response Status: successful (0x0) Response Type: Basic OCSP Response Version: 1 (0x0) ...

Details of the OCSP Stapling TLS extension can be found in RFC 6961. To configure Traffic Server to use OCSP Stapling, edit the following variables in records.config file: • :ts:cv:‘proxy.config.ssl.ocsp.enabled‘ • :ts:cv:‘proxy.config.ssl.ocsp.cache_timeout‘ • :ts:cv:‘proxy.config.ssl.ocsp.request_timeout‘ • :ts:cv:‘proxy.config.ssl.ocsp.update_period‘

Split DNS

The Split DNS option enables you to configure Traffic Server to use multiple DNS servers, as dictated by your security requirements. For example, you might configure Traffic Server to use one set of DNS servers to resolve hostnames on your internal network, while allowing DNS servers outside the firewall to resolve hosts on the Internet. This maintains the security of your intranet, while continuing to provide direct access to sites outside your organization. To configure Split DNS: 1. Specify the rules for performing DNS server selection based on the destination domain, the destination host, or a URL regular expression. These rules are located in splitdns.config. 2. Enable the Split DNS option by adjusting :ts:cv:‘proxy.config.dns.splitDNS.enabled‘ in records.config.

CONFIG proxy.config.dns.splitDNS.enabled INT1

3. Run the command traffic_line -x to apply the conﬁguration changes.

Cache Storage

The Trafﬁc Server cache consists of a high-speed object database called the object store that indexes cache objects according to URLs and their associated headers.

The Trafﬁc Server Cache

The Traffic Server cache consists of a high-speed object database called the object store. The object store indexes cache objects according to URLs and associated headers. This enables Traffic Server to store, retrieve, and serve not only web pages, but also parts of web pages - which provides optimum bandwidth savings. Using sophisticated object management, the object store can cache alternate versions of the same object (versions may differ because of dissimilar language or encoding types). It can also efficiently store very small and very large documents, thereby minimizing wasted space. When the cache is full, Traffic Server removes stale data to ensure the most requested objects are kept readily available and fresh.

58 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Traffic Server is designed to tolerate total disk failures on any of the cache disks. If the disk fails completely, then Traffic Server marks the entire disk as corrupt and continues using the remaining disks. An alarm is then created to indicate which disk failed. If all of the cache disks fail, then Traffic Server goes into proxy-only mode. You can perform the following cache configuration tasks: • Change the total amount of disk space allocated to the cache; refer to Changing Cache Capacity. • Partition the cache by reserving cache disk space for specific protocols and origin servers/domains; refer to Partitioning the Cache. • Delete all data in the cache; refer to Clearing the Cache. • Override cache directives for a requested domain name, regex on a url, hostname or ip, with extra filters for time, port, method of the request, and more. ATS can be configured to never cache, always cache, ignore no-cache directives, etc. These are configured in cache.config.

The RAM Cache

Traffic Server maintains a small RAM cache of extremely popular objects. This RAM cache serves the most popular objects as quickly as possible and reduces load on disks, especially during temporary traffic peaks. You can configure the RAM cache size to suit your needs, as described in Changing the Size of the RAM Cache below. The RAM cache supports two cache eviction algorithms, a regular LRU (Least Recently Used) and the more advanced CLFUS (Clocked Least Frequently Used by Size; which balances recentness, frequency, and size to maximize hit rate, similar to a most frequently used algorithm). The default is to use CLFUS, and this is controlled via :ts:cv:‘proxy.config.cache.ram_cache.algorithm‘. Both the LRU and CLFUS RAM caches support a configuration to increase scan resistance. In a typical LRU, if you request all possible objects in sequence, you will effectively churn the cache on every request. The option :ts:cv:‘proxy.config.cache.ram_cache.use_seen_filter‘ can be set to add some resistance against this problem. In addition, CLFUS also supports compressing in the RAM cache itself. This can be useful for content which is not compressed by itself (e.g. images). This should not be confused with Content-Encoding: gzip, this feature is only present to save space internally in the RAM cache itself. As such, it is completely transparent to the User-Agent. The RAM cache compression is enabled with the option :ts:cv:‘proxy.config.cache.ram_cache.compress‘. Possible values are: Value Meaning 0 No compression (default) 1 fastlz compression 2 libz compression 3 liblzma compression

Changing the Size of the RAM Cache

Trafﬁc Server provides a dedicated RAM cache for fast retrieval of popular small objects. The default RAM cache size is automatically calculated based on the number and size of the cache partitions you have conﬁgured. If you’ve partitioned your cache according to protocol and/or hosts, then the size of the RAM cache for each partition is proportional to the size of that partition. You can increase the RAM cache size for better cache hit performance. However, if you increase the size of the RAM cache and observe a decrease in performance (such as increased latencies), then it’s possible that the operating system requires more memory for network resources. In such instances, you should return the RAM cache size to its previous value. To change the RAM cache size:

3.6. Cache Storage 59 Apache-Trafﬁcserver-Server Documentation, latest

1. Stop Traffic Server. 2. Set the variable :ts:cv:‘proxy.config.cache.ram_cache.size‘ to specify the size of the RAM cache. The default value of -1 means that the RAM cache is automatically sized at approximately 1MB per gigabyte of disk. 3. Restart Traffic Server. If you increase the RAM cache to a size of 1GB or more, then restart with the trafficserver command (refer to Start Traffic Server).

Changing Cache Capacity

You can increase or reduce the total amount of disk space allocated to the cache without clearing the content. To check the size of the cache (in bytes), enter the command traffic_ctl metric get proxy.process.cache. bytes_total.

Increasing Cache Capacity

To increase the total amount of disk space allocated to the cache on existing disks, or to add new disks to a Traffic Server node: 1. Stop Traffic Server. 2. Add hardware, if necessary. 3. Edit storage.config to increase the amount of disk space allocated to the cache on existing disks or describe the new hardware you are adding. 4. Restart Traffic Server.

Reducing Cache Capacity

To reduce the total amount of disk space allocated to the cache on an existing disk, or to remove disks from a Traffic Server node: 1. Stop Traffic Server. 2. Remove hardware, if necessary. 3. Edit storage.config to reduce the amount of disk space allocated to the cache on existing disks or delete the reference to the hardware you’re removing. 4. Restart Traffic Server.

: In storage.config, a formatted or raw disk must be at least 128 MB.

Partitioning the Cache

You can manage your cache space more efficiently and restrict disk usage by creating cache volumes with different sizes for specific protocols. You can further configure these volumes to store data from specific origin servers and/or domains. The volume configuration must be the same on all nodes in a cluster.

60 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Creating Cache Partitions for Speciﬁc Protocols

You can create separate volumes for your cache that vary in size to store content according to protocol. This ensures that a certain amount of disk space is always available for a particular protocol. Trafﬁc Server currently supports only the http partition type. To partition the cache according to protocol: 1. Enter a line in volume.config for each volume you want to create.

volume=1 scheme=http size=50% volume=2 scheme=http size=50%

2. Restart Trafﬁc Server.

: Volume deﬁnitions must be the same across all nodes in a cluster.

Making Changes to Partition Sizes and Protocols

After you’ve configured your cache volumes based on protocol, you can make changes to the configuration at any time. Before making changes, note the following: • You must stop Traffic Server before you change the cache volume size and protocol assignment. • When you increase the size of a volume, the contents of the volume are not deleted. However, when you reduce the size of a volume, the contents of the volume are deleted. • When you change the volume number, the volume is deleted and then recreated, even if the size and protocol type remain the same. • When you add new disks to your Traffic Server node, volume sizes specified in percentages will increase pro- portionately. • Substantial changes to volume sizes can result in disk fragmentation, which affects performance and cache hit rate. You should clear the cache before making many changes to cache volume sizes (refer to Clearing the Cache).

Partitioning the Cache According to Origin Server or Domain

After you have partitioned the cache according to size and protocol, you can assign the volumes you created to specific origin servers and/or domains. You can assign a volume to a single origin server or to multiple origin servers. However, if a volume is assigned to multiple origin servers, then there is no guarantee on the space available in the volumes for each origin server. Content is stored in the volumes according to popularity. In addition to assigning volumes to specific origin servers and domains, you must assign a generic volume to store content from all origin servers and domains that are not listed. This generic volume is also used if the partitions for a particular origin server or domain become corrupt. If you do not assign a generic volume, then Traffic Server will run in proxy-only mode.

: You do not need to stop Traffic Server before you assign volumes to particular hosts or domains. However, this type of configuration is time-consuming and can cause a spike in memory usage. Therefore, it’s best to configure partition assignment during periods of low traffic.

To partition the cache according to hostname and domain:

3.6. Cache Storage 61 Apache-Trafﬁcserver-Server Documentation, latest

1. Configure the cache volumes according to size and protocol, as described in Creating Cache Partitions for Specific Protocols. 2. Create a separate volume based on protocol for each host and domain, as well as an additional generic partition to use for content that does not belong to these origin servers or domains. The volumes do not need to be the same size. 3. Enter a line in the hosting.config file to allocate the volume(s) used for each origin server and/or domain. 4. Assign a generic volume to use for content that does not belong to any of the origin servers or domains listed in the file. If all volumes for a particular origin server become corrupt, then Traffic Server will also use the generic volume to store content for that origin server as per hosting.config. 5. Run the command traffic_ctl config reload to apply the configuration changes.

Conﬁguring the Cache Object Size Limit

By default, Traffic Server allows objects of any size to be cached. You can change the default behavior and specify a size limit for objects in the cache via the steps below: 1. Set :ts:cv:‘proxy.config.cache.max_doc_size‘ to specify the maximum size in bytes allowed for objects in the cache. A setting of 0 (zero) will permit cache objects to be unlimited in size. 2. Run the command traffic_ctl config reload to apply the configuration changes.

Clearing the Cache

When you clear the cache, you remove all data from the entire cache - including data in the host database. You should clear the cache before performing certain cache configuration tasks such as partitioning. You cannot clear the cache when Traffic Server is running. To clear the cache: 1. Stop Traffic Server (see Stop Traffic Server) 2. Enter the following command to clear the cache:

traffic_server-Cclear

The clear command deletes all data in the object store and the host database. Traffic Server does not prompt you to confirm the deletion. 3. Restart Traffic Server (see Start Traffic Server).

Removing an Object From the Cache

Traffic Server accepts the custom HTTP request method PURGE when removing a specific object from cache. If the object is found in the cache and is successfully removed, then Traffic Server responds with a 200 OK HTTP message; otherwise, a 404 File Not Found message is returned.

: By default, the PURGE request method is only processed if received on the localhost interface.

In the following example, Trafﬁc Server is running on the domain example.com and you want to remove the image remove_me.jpg from cache. Because by default we do not permit PURGE requests from any other IP, we connect to the daemon via localhost:

62 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

$ curl -vX PURGE --resolve example.com:80:127.0.0.1 http://example.com/remove_me.jpg * About to connect() to example.com port 80 (#0) * Trying 127.0.0.1... connected * Connected to example.com (127.0.0.1) port 80 (#0)

> PURGE /remove_me.jpg HTTP/1.1 > User-Agent: curl/7.19.7 > Host: example.com > Accept: */* > < HTTP/1.1 200 Ok < Date: Thu, 08 Jan 2010 20:32:07 GMT < Connection: keep-alive

The next time Traffic Server receives a request for the removed object, it will contact the origin server to retrieve a new copy, which will replace the previously cached version in Traffic Server. This procedure only removes an object from a specific Traffic Server cache. Users may still see the old (removed) content if it was cached by intermediary caches or by the end-users’ web browser.

Inspecting the Cache

Trafﬁc Server provides a Cache Inspector utility that enables you to view, delete, and invalidate URLs in the cache (HTTP only). The Cache Inspector utility is a powerful tool that’s capable of deleting all the objects in your cache. Therefore, make sure that only authorized administrators are allowed to access this utility through proper use of the @src_ip option in remap.config and the instructions detailed in Controlling Access.

Accessing the Cache Inspector Utility

To access the Cache Inspector utility: 1. Set :ts:cv:‘proxy.conﬁg.http_ui_enabled‘ to 1. 2. To access the cache inspector in reverse proxy mode, you must add a remap rule to remap.config to expose the URL. This should be restricted to a limited set of hosts using the @src_ip option. To restrict access to the network 172.28.56.0/24, use

map http://yourhost.com/myCI/ http://{cache} @action=allow @src_ip=172.28.56.1-

˓→172.28.56.254

3. Reload the Trafﬁc Server conﬁguration by running traffic_ctl config reload. 4. Open your web browser and go to the the following URL:

http://yourhost/myCI/

You will now be presented with the Cache Inspector interface.

Using the Cache Inspector Utility

The Cache Inspector Utility provides several options that enable you to view and delete the contents of your cache. Lookup URL Search for a particular URL in the cache. When Trafﬁc Server ﬁnds the URL in the cache, it will display details of the object that corresponds to the URL (e.g. header length and number of alternates). The option to delete the URL from the cache will be presented.

3.6. Cache Storage 63 Apache-Trafﬁcserver-Server Documentation, latest

Delete URL Delete the object from the cache which corresponds to the given URL. Success or failure will be indicated after a delete has been attempted. Regex Lookup Search URLs within the cache using one or more regular expressions. Regex Delete Deletes all objects from the cache which match the provided regular expressions. Regex Invalidate Marks any objects in the cache which match the given regular expressions as stale. Trafﬁc Server will contact the relevant origin server(s) to conﬁrm the validity and freshness of the cached object, updating the cached object if necessary.

: Only one administrator should delete and invalidate cache entries from the Cache Inspector at any point in time. Changes made by multiple administrators at the same time can lead to unpredictable results.

Plugins

Overview

One of the key features of Apache Trafﬁc Server™ is its modularity. Features that aren’t needed in the core simply aren’t there. This helps to provide an additional guarantee that our core can remain fast by concentrating on the things that we always provide: caching and proxying. All other functionality can be moved into plugins and by offering a consistent C API, everyone can implement their own functionality, without having to touch the core.

Stable plugins

Plugins that are considered stable are installed by default in Trafﬁc Server releases.

Cache URL Plugin

: This plugin is deprecated as of v6.2.0 and will be removed as of v7.0.0. It is replaced by a new Cache Key Manipulation Plugin and you should change your conﬁgurations to use the new plugin instead. Please ﬁnd some examples below.

This plugin allows you to change the cache key that is used for caching a request by using any portion of the URL via regular expressions.

Purpose

By default Traffic Server generates keys for cache objects from the full request URL. Future requests will only be able to use the existing cache object if they result in the same cache key. It can be the case, however, that the same content is accessible through more than one request URL. Without any special configuration, that same content will result in multiple cache objects based on the differing URLs. In this scenario, it can be very beneficial to allow Traffic Server to reuse a cache object for multiple URLs. This plugin enables that behavior, by allowing the administrator to define custom patterns for generating cache keys for some, or all, of their site’s URLs. Uninteresting or irrelevant portions of request URLs may be removed, or altered, before the cache key is created using the full power of regular expressions.

64 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Installation

This plugin is considered stable and is included with Trafﬁc Server by default. There are no special steps necessary for its installation.

Conﬁguration

1. Enable the plugin by modifying plugin.config to include the plugin:

cacheurl.so

2. Create a cacheurl.config ﬁle in the plugin directory with the url regex patterns you wish to match, using the following format:

The is a regular expression (PCRE) applied against the incoming request URL. The may contain $1, $2, and so on, which will be replaced with the appropriate matching group from . 3. Reload your Trafﬁc Server conﬁguration with traffic_ctl config reload.

Logging

A new log ﬁle will be generated by this plugin, containing entries for each incident of an incoming URL’s cache key being altered, located in the Trafﬁc Server log directory and named cacheurl.log.

Examples

While many possibilities exist, limited really only by your site’s URL scheme and the capabililties of PCRE regular expressions, the following are examples of a few situations Trafﬁc Server administrators may encounter.

Multiple Domains, One Cache Object

If you have multiple subdomains which serve the same ﬁle content, there may be no reason to duplicate their storage (leading to higher churn and faster potential eviction of still-fresh objects) in your Trafﬁc Server cache. By default, however, the differing subdomains will lead to differing cache keys. To work around this, a pattern like the following can be used to create a single cache key which will be valid for all subdomains: http://s[123].example.com/(.*) http://s.example.com.TSINTERNAL/$1

Now, the domains s1.example.com, s2.example.com, and s3.example.com will effectively share cache objects. Adding a unique sufﬁx (TSINTERNAL in this example) to the cache key guarantees that it won’t clash with a real URL should s.example.com exist.

Converting to Cache Key Manipulation Plugin

You could do the same with Cache Key Manipulation Plugin by adding the following to the mapping rules:

3.7. Plugins 65 Apache-Trafﬁcserver-Server Documentation, latest

@plugin=cachekey.so @pparam=--capture-prefix=/s[123].example.com:.*/s.example.com. ˓→TSINTERNAL/

Ignoring Some Query Parameters

If your site attaches, for example, session information to URLs, even on pages which do not include session-speciﬁc content and may be safely cached, you can add a pattern which strips this unnecessary information from the URL before generating a cache key, while still retaining important query parameters:

http://www.example.com/video\?.*?\&?(id=[0-9a-f]*).*?\&(format=[a-z]*) http://video- ˓→srv.example.com.ATSINTERNAL/$1&$2

Converting to Cache Key Manipulation Plugin

You could do the same with Cache Key Manipulation Plugin by adding the following to the mapping rules:

@plugin=cachekey.so @pparam=--include-params=id,format

Ignore Query String on Speciﬁc Pages

To completely ignore a query string for a speciﬁc page, it’s quite easy to simply match and drop everything from the ? query string opener to the end of the URL:

http://www.example.com/some/page(?:\?|$) http://www.example.com/some/page

Converting to Cache Key Manipulation Plugin

You could do the same with Cache Key Manipulation Plugin by adding the following to the mapping rules:

@plugin=cachekey.so @pparam=--remove-all-params

Conﬁguration Remap Plugin

This plugin allows you to override conﬁguration directives dependent on remapping rules.

Purpose

Traffic Server provides a plethora of configuration options, but specifying the values of those options in records. config is global. All requests, regardless of the cache object or its origin, will be evaluated within the same collection of settings. Sometimes you may want Traffic Server to behave differently for portions of your cache. Perhaps you have Negative Response Caching enabled, but you wish to greatly reduce the validity times for just one of your origin servers while allowing the rest of your origins to have their errors cached for long durations. Or maybe you make use of Heuristic Expiration but require different fuzz times for various objects because of the nature of their content and expected lifetimes.

66 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Any configuration directive which is overridable can be modified on a per-map basis with this plugin. This opens up a level of flexibility in your configurations for effectively managing and caching content with varied needs, without having to resort to multiple Traffic Server instances.

Installation

This plugin is considered stable and is included with Trafﬁc Server by default. There are no special steps necessary for its installation.

Conﬁguration

Configuration of this plugin is performed alongside the actual remapping rules which trigger the desired configuration directive changes. There are two methods available for specifying the actual directives and their modified values.

Inline Directives

In cases where you have very few remapping rules which modify directives, and they are modifying only a small number of directives, you may find it easiest to simply specify those directive changes in-line with your remapping rules. This is done by specifying key = value pairs, where the key is the configuration directive name and the value is its desired setting for the remapping rule. For example, the enable :ts:cv:‘proxy.config.url_remap.pristine_host_hdr‘ for a single map rule, you would add the following to your remap.config:

map http://cdn.example.com/ http://origin.example.com/\ @plugin=conf_remap.so @pparam=proxy.config.url_remap.pristine_host_hdr=1

External Conﬁguration

There may be situations in which you have many directives you wish to modify, or where multiple remapping rules perform the same directive changes. External configurations can simplify management of these rules, and help to reduce the possibility of transcription errors, or keeping all the directive settings in sync across all the remapping rules over time. Instead of specifying the directives and their values in remap.config as you do with the in-line method, you place all the affected directives in a separate text file. The location and name is entirely up to you, but we’ll use /etc/trafficserver/cdn_conf_remap.config here. The contents of this file should mirror how configuration directives are written in records.config:

CONFIG proxy.config.url_remap.pristine_host_hdr INT1

Your remap.config will then contain remapping rules that point to this external ﬁle:

map http://cdn.example.com/ http://some-server.example.com \ @plugin=conf_remap.so @pparam=/etc/trafficserver/cdn_conf_remap.config

Your external conﬁguration may contain as many directives as you wish.

3.7. Plugins 67 Apache-Trafﬁcserver-Server Documentation, latest

Caveats

This plugin can only modify the values for those configuration directives which are overridable, meaning they are not fixed upon Traffic Server startup. While this generally shouldn’t prove too onerous a restriction, you should consult the individual directives’ documentation to confirm whether they may be overridden.

Further Reading

For more information about the implementation of overridable conﬁguration directives, you may consult the developer’s documentation for TSHttpOverridableConﬁg.

GZip Plugin

This plugin adds compression and decompression options to both origin and cache responses.

Purpose

Not all clients can handle compressed content. Not all origin servers are configured to respond with compressed content when a client says it can accept it. And it’s not always necessary to make two separate requests to an origin, and track two separate cache objects, for the same content - once for a compressed version and another time for an uncompressed version. This plugin tidies up these problems by transparently compressing or deflating origin responses, as necessary, so that both variants of a response are stored as alternates and the appropriate version is used for client responses, depending on the client’s indication (via an Accept request header) of what it can support. Additionally, this plugin adds configurability for what types of origin responses will receive this treatment, which will be proxied and cached with default behavior, and which may be explicitly disallowed to cache both compressed and deflated versions (because, for example, the cost of compression is known ahead of time to outweigh the space and bandwidth savings and you wish to avoid Traffic Server even testing for the possibility).

Installation

This plugin is considered stable and is included with Trafﬁc Server by default. There are no special steps necessary for its installation.

Conﬁguration

This plugin can be used as either global plugin or remap plugin. It can be enabled globally for Trafﬁc Server by adding the following to your plugin.config:

gzip.so

With no further options, this will enable the following default behavior: • Enable caching of both compressed and uncompressed versions of origin responses as alternates. • Compress objects with text/* content types for every origin. • Don’t hide Accept encoding headers from origin servers (for an ofﬂoading reverse proxy). • No URLs are disallowed from compression.

68 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

• Disable flush (flush gzipped content to client). Alternatively, a configuration may be specified (shown here using the sample configuration provided with the plugin’s source):

gzip.so/sample.gzip.config

This can be used as remap plugin by pointing to conﬁg ﬁle in remap rule remap.config:

@plugin=gzip.so @pparam=--config=/sample.gzip.config

The following sections detail the options you may specify in the plugin’s configuration file. Options may be used globally, or may be specified on a per-site basis by preceding them with a [] line, where is the client- facing domain for which the options should apply. Per site configuration for remap plugin should be ignored.

cache

When set to true, causes Trafﬁc Server to cache both the compressed and uncompressed versions of the content as alternates. When set to false, Trafﬁc Server will cache only the compressed or decompressed variant returned by the origin. Enabled by default.

compressible-content-type

Provides a wildcard to match against content types, determining which are to be considered compressible. This defaults to text/*.

disallow

Provides a wildcard pattern which will be applied to request URLs. Any which match the pattern will be considered uncompressable, and only deﬂated versions of the objects will be cached and returned to clients. This may be useful for objects which already have their own compression built-in, to avoid the expense of multiple rounds of compression for trivial gains.

enabled

When set to true permits objects to be compressed, and when false effectively disables the plugin in the current context.

ﬂush

Enables (true) or disables (false) ﬂushing of compressed objects to clients.

remove-accept-encoding

When set to true this option causes the plugin to strip the request’s Accept encoding header when contacting the origin server. Setting this option to false will leave the header intact if the client provided it. • To ease the load on the origins.

3.7. Plugins 69 Apache-Trafﬁcserver-Server Documentation, latest

• For when the proxy parses responses, and the resulting compression and decompression is wasteful.

Examples

To establish global defaults for all site requests passing through Trafﬁc Server, while overriding just a handful for requests to content at www.example.com, you might create a conﬁguration with the following options:

# Set some global options first cache true enabled true remove-accept-encoding false compressible-content-type text/* flush false

# Now set a configuration for www.example.com [www.example.com] cache false remove-accept-encoding true disallow/notthis/ *.js flush true

Assuming the above options are in a ﬁle at /etc/trafficserver/gzip.config the plugin would be enabled for Trafﬁc Server in plugin.config as: gzip.so/etc/trafficserver/gzip.config

Header Rewrite Plugin

This plugin allows you to modify arbitrary headers based on deﬁned rules, for both requests and responses.

Purpose

Remapping an incoming client request to an origin server is at the heart of what we use Traffic Server for, but often we need to do more to requests than just change their destination according to simple rewriting rules against the URL. We may need to direct requests to different origins based on a client cookie. Our origins might return error response codes which are a little too customized and we want to condense the possible values to just the official codes. We might want to strip a set of internal-use or debugging related HTTP headers from responses before sending them to clients, unless the original request had its own special header indicating that they should be retained. Or perhaps we want to redirect certain requests differently when they come from a known group of IP addresses (say, our developers’ office network) and we have a file on our proxy caching servers called /var/developertesting. (Stranger QA methods exist.) Maybe we want to deny access to a resource with an HTTP 403 if the client connected to Traffic Server over a particular port, used HEAD instead of GET, doesn’t list Spanish (Paraguay dialect) in their Accept-Language header, and either the origin server replied with 304 or we randomly generate an integer between 0 and 500 and get back anything greater than 290. These more complicated transformations of requests and responses (even that last one) are made possible by this plugin.

70 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Installation

This plugin is considered stable and is included with Trafﬁc Server by default. There are no special steps necessary for its installation.

Conﬁguration

Header rewrite configurations, the actual conditions and operations that make up the activity performed by the plugin, are specified in external files and not in-line with the various mapping (and remapping) rules you may have configured for your proxy. The location of this file is arbitrary, as long as the Traffic Server processes have permissions to read it, though you may find it useful to keep it in the same location as your other proxy configuration files. The paths given to the configuration file(s) may be absolute (leading with a / character), or they may be relative to the Traffic Server configuration directory. There are two methods for enabling this plugin, based on whether you wish it to operate globally on every request that passes through your proxy, or only on some subset of the requests by enabling it only for specific mapping rules.

Enabling Globally

This plugin may be enabled globally, so that the conditions and header rewriting rules are evaluated for every request made to your Trafﬁc Server instance. This is done by adding the following line to your plugin.config:

header_rewrite.so config_file_1.conf config_file_2.conf...

You may specify multiple configuration files. Their rules will be evaluated in the order the files are listed.

Enabling Per-Mapping

Alternatively, the plugin can be enabled for speciﬁc mapping rules, by modifying the relevant entries in your remap. config:

map http://a http://b @plugin=header_rewrite.so @pparam=rules1.conf...

As with the global method above, multiple configuration files may be listed, each with its own @pparam= and their contents will be evaluated in the order the files were specified.

Rewriting Rules

Header rewriting rules consist of zero or more Conditions followed by one or more Operators. Conditions are used to limit the requests which will be affected by the operator(s). Additionally, both conditions and operators may have ﬂags which modify their behavior. A complete rule, consisting of two conditions and a single operator might look like the following:

cond%{STATUS}>399 [AND] cond%{STATUS}<500 set-status 404

Which converts any 4xx HTTP status code from the origin server to a 404. A response from the origin with a status of 200 would be unaffected by this rule.

3.7. Plugins 71 Apache-Trafﬁcserver-Server Documentation, latest

Conditions

Conditions are used as qualiﬁers, causing the associated operators to only be evaluated if the condition(s) are met. Conditions all take the following form:

cond%{[:]}[]

Every condition begins with the literal string cond to indicate that this line is a condition, not an operator. This is followed by the condition name, inside curly braces and preceded by a percent sign (e.g. %{TRUE} for the condition named TRUE). Some condition names take an argument. Header conditions, for example, take the name of the header in question, and cookie conditions take the name of the cookie. For these, the condition name is followed by a colon and the argument value (e.g. %{HEADER:User-Agent} for a header condition against the User-Agent header). The operand of a condition provides a value, pattern, or range against which to match. The format is described in Condition Operands below. Finally, a condition may optionally have various ﬂags associated with it. These are described in Condition Flags below. The following sections list all of the condition types currently supported. For increased clarity in their usage, the optional [] portion of the condition is omitted from all of the examples.

ACCESS

cond%{ACCESS:}

Returns true if Traffic Server was able to successfully update the access time on the file at . This condition will return false if the file does not exist or Traffic Server cannot access it for any other reason.

CLIENT-HEADER

cond%{CLIENT-HEADER:}

Value of the header from the original client request (regardless of the hook context in which the rule is being evaluated). Note that some headers may appear in an HTTP message more than once. In these cases, the value of the header operated on by this condition will be a comma separated string of the values from every occurrence of the header. More details are provided in Repeated Headers below.

CLIENT-IP

cond%{CLIENT-IP}

Remote IP address, as a string, of the client connection for the current transaction.

CLIENT-URL

cond%{CLIENT-URL:}

The URL of the original request. Regardless of the hook context in which the rule is evaluated, this condition will always operate on the original, unmapped URL supplied by the client. The may be speciﬁed according to the options documented in URL Parts.

72 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

COOKIE cond%{COOKIE:}

Value of of the cookie . This does not expose or match against a cookie’s expiration, the domain(s) on which it is valid, whether it is protocol restricted, or any of the other metadata; simply the current value of the cookie as presented by the client.

FROM-URL cond%{FROM-URL:}

In a remapping context, this condition matches against the source URL from which the remapping was generated. This condition is valid only within conﬁgurations provided through remap.config as described in Enabling Per- Mapping above. The allows the operand to match against just a component of the URL, as documented in URL Parts below.

GEO

:: cond %{GEO:} Perform a GeoIP lookup of the client-IP, using a 3rd party library and DB. Currently only the MaxMind GeoIP API is supported. The default is to do a Country lookup, but the following qualiﬁers are supported: %{GEO:COUNTRY} The country code (e.g. “US”) %{GEO:COUNTRY-ISO} The country ISO code (e.g. 225) %{GEO:ASN} The AS number for the provider network (e.g. 7922) %{GEO:ASN-NAME} A descriptive string of the AS provider These operators can be used both as conditionals, as well as values for setting headers. For example: cond %{SEND_RESPONSE_HDR_HOOK} [AND] cond %${GEO:COUNTRY} =US set-header ATS-Geo-Country %{GEO:COUNTRY} set-header ATS-Geo-Country-ISO %{GEO:COUNTRY-ISO} set-header ATS-Geo-ASN %{GEO:ASN} set-header ATS-Geo-ASN-NAME %{GEO:ASN-NAME}

HEADER cond%{HEADER:}

Value of the header from either the original client request or the origin server’s response, depending upon the hook context in which the rule is being evaluated. Consult Requests vs. Responses for more information on how to distinguish the two, as well as enforce that a rule is always evaluated in the desired context. Note that some headers may appear in an HTTP message more than once. In these cases, the value of the header operated on by this condition will be a comma separated string of the values from every occurrence of the header. Refer to Repeated Headers for more information. If you wish to use a client request header, regardless of hook context, you may consider using the CLIENT-HEADER condition instead.

3.7. Plugins 73 Apache-Trafﬁcserver-Server Documentation, latest

INCOMING-PORT

cond%{INCOMING-PORT}

TCP port, as a decimal integer, on which the incoming client connection was made.

INTERNAL-TRANSACTION

cond%{INTERNAL-TRANSACTION}

Returns true if the current transaction was internally-generated by Trafﬁc Server (using TSHttpTxnIsInternal()). These transactions are not initiated by external client requests, but are triggered (often by plugins) entirely within the Trafﬁc Server process.

METHOD

cond%{METHOD}

The HTTP method (e.g. GET, HEAD, POST, and so on) used by the client for this transaction.

NOW

cond%{NOW:}

This is the current time, in the local timezone as set on the machine, typically GMC. Without any further qualiﬁers, this is the time in seconds since EPOCH aka Unix time. Qualiﬁers can be used to give various other values, such as year, month etc.

%{NOW:YEAR} Current year (e.g. 2016) %{NOW:MONTH} Current month (0-11,0 == January) %{NOW:DAY} Current day of the month (1-31) %{NOW:HOUR} Current hour (0-23, in the 24h system) %{NOW:MIN} Current minute (0-59} %{NOW:WEEKDAY} Current weekday (0-6,0 == Sunday) %{NOW:YEARDAY} Current day of the year (0-365,0 == Jan1st)

PATH

cond%{PATH}

The path component of the transaction. This includes the leading / that immediately follows the hostname and terminates prior to the ? signifying the beginning of query parameters (or the end of the URL, whichever occurs ﬁrst). Refer to Requests vs. Responses for more information on determining the context in which the transaction’s URL is evaluated.

74 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

QUERY

cond%{QUERY}

The query parameters, if any, of the transaction. Refer to Requests vs. Responses for more information on determining the context in which the transaction’s URL is evaluated.

RANDOM

cond%{RANDOM:}

Generates a random integer between 0 and , inclusive.

STATUS

cond%{STATUS}

Numeric HTTP status code of the response.

TO-URL

cond%{TO-URL:}

In a remapping context, this condition matches against the target URL to which the remapping is directed. This condition is valid only within conﬁgurations provided through remap.config as described in Enabling Per-Mapping above. The allows the operand to match against just a component of the URL, as documented in URL Parts below.

TRUE / FALSE

cond%{TRUE} cond%{FALSE}

These conditions always return a true value and a false value, respectively. The true condition is implicit in any rules which specify no conditions (only operators).

TXN-COUNT

cond%{TXN-COUNT}

Returns the current HTTP client session, which permits detection of requests which are sharing a client session. Shared client sessions occur when multiple simultaneous requests are received for the same cache object. Instead of contacting the origin server separately for each of those client requests, one origin connection is used to fulﬁll all of the requests assigned to the shared client session.

3.7. Plugins 75 Apache-Trafﬁcserver-Server Documentation, latest

URL cond%{URL:option}

The complete URL of the current transaction. This will automatically choose the most relevant URL depending upon the hook context in which the condition is being evaluated. Refer to Requests vs. Responses for more information on determining the context in which the transaction’s URL is evaluated.

Condition Operands

Operands provide the means to restrict the values, provided by a condition, which will lead to that condition evaluating as true. There are currently four types supported: Operand Description /regex/ Matches the condition’s provided value against the regular expression. string Matches if the value from the condition is lexically greater than string. =string Matches if the value from the condition is lexically equal to string. The absence of an operand for conditions which accept them simply requires that a value exists (e.g. the content of the header is not an empty string) for the condition to be considered true.

Condition Flags

The condition flags are optional, and you can combine more than one into a comma-separated list of flags. Note that whitespaces are not allowed inside the brackets: Flag Description AND Indicates that both the current condition and the next must be true. This is the default behavior for all conditions when no flags are provided. NOT Inverts the condition. OR Indicates that either the current condition or the next one must be true, as contrasted with the default behavior from [AND].

Operators

Operators are the part of your header rewriting rules which actually modify the header content of your requests and responses. They are always the final part of a rule, following any of the conditions which whittled down the requests and responses to which they will be applied. Multiple operators may be specified for a single rule, and they will be executed in the order listed. The end of the rule is marked either by the end of the configuration file or the next appearance of a condition (whichever occurs first). The following operators are available: add-header add-header

76 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Adds a new header line with the contents . Note that this operator can produce duplicate headers if one of already exists, or your configuration supplies multiple instances of this operator in different rules which are all invoked. This is not an issue for headers which may safely be specified multiple times, such as Set-Cookie, but for headers which may only be specified once you may prefer to use set-header instead. The header’s may be specified as a literal string, or it may take advantage of Variable Expansion to calculate a dynamic value for the header. In contrast, set-header does not support variable expansion for the header value. If you wish to use variable expansion and avoid duplicate headers, you may consider using an rm-header operator followed by add-header.

counter

Increments an integer counter called every time the rule is invoked. The counter is initialized at 0 if it does not already exist. The name you give your counter is arbitrary, though it is strongly advisable to avoid conflicts with existing Traffic Server statistics. This counter can be viewed at any time through the standard statistics APIs, including the Stats Over HTTP plugin. Counters can only increment by 1 each time this operator is invoked. There is no facility to increment by other amounts, nor is it possible to initialize the counter with any value other than 0. Additionally, the counter will reset whenever Traffic Server is restarted.

no-op

This operator does nothing, takes no arguments, and has no side effects.

rm-header

Removes the header .

set-conﬁg

set-config

Allows you to override the value of a Traffic Server configuration variable for the current connection. The variables specified by must be overridable. For details on available Traffic Server configuration variables, consult the documentation for records.config. You can read more about overridable configuration variables in the developer’s documentation for TSHttpOverridableConfig.

set-conn-dscp

3.7. Plugins 77 Apache-Trafﬁcserver-Server Documentation, latest

When invoked, sets the client side DSCP value for the current transaction. The should be speciﬁed as a decimal integer.

set-debug

When invoked, this operator enables the internal transaction debugging ﬂag (via TSHttpTxnDebugSet()), which causes debug messages to be printed to the appropriate logs even when the debug tag has not been enabled. For additional information on Trafﬁc Server debugging statements, refer to Debug Tags in the developer’s documentation.

set-destination

Modiﬁes individual components of the remapped destination’s address. When changing the remapped destination, should be used to indicate the component that is being modiﬁed (see URL Parts), and will be used as its replacement. You must supply a non-zero length value, otherwise this operator will be an effective no-op (though a warning will be emitted to the logs if debugging is enabled). set-header

set-header

Replaces the value of header with , creating the header if necessary. Note that, unlike add-header, this operator does not currently support variable expansion. Values may only be speciﬁed according to Header Values.

set-redirect

When invoked, sends a redirect response to the client, with HTTP status , and a new location of . If the QSA ﬂag is enabled, the original query string will be preserved and added to the new location automatically. This operator supports Variable Expansion for . set-status

set-status

Modiﬁes the HTTP status code used for the response. must be a valid status code. This operator will also update the reason in the HTTP response, based on the code you have chosen. If you wish to override that with your own text, you will need to invoke the set-status-reason operator after this one.

78 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

set-status-reason

Modiﬁes the HTTP response to use as the HTTP status reason, instead of the standard string (which depends on the HTTP status code used). set-timeout-out

set-timeout-out

Modiﬁes the timeout values for the current transaction to (speciﬁed in milliseconds). The must be one of the following: active, inactive, connect, or dns.

skip-remap

When invoked, and when is any of 1, true, or TRUE, this operator causes Trafﬁc Server to abort further request remapping. Any other value and the operator will effectively be a no-op.

Operator Flags

Operator ﬂags are optional, are separated by commas when using more than one for a single operator, and must not contain whitespace inside the brackets. For example, an operator with the L ﬂag would be written in this manner:

set-destination HOST foo.bar.com [L]

The ﬂags currently supported are: Flag Description L Last rule, do not continue. QSA Append the results of the rule to the query string.

Variable Expansion

Only limited variable expansion is supported in add-header. Supported substitutions are currently: Variable Description % Protocol % Port % Client IP % Client request length % Client HTTP method % Client unmapped URI

Header Values

Setting a header with a value can take the following formats:

3.7. Plugins 79 Apache-Trafﬁcserver-Server Documentation, latest

• Any condition which extracts a value from the request. • $N, where 0 <= N <= 9, from matching groups in a regular expression. • A string (which can contain the numbered matches from a regular expression as described above). • Null. Supplying no value for a header for certain operators can lead to an effective no-op. In particular, add-header and set-header will simply short-circuit if no value has been supplied for the named header.

URL Parts

Some of the conditions and operators which use a request or response URL as their target allow for matching against speciﬁc components of the URL. For example, the CLIENT-URL condition can be used to test just against the query parameters by writing it as: cond%{CLIENT-URL:QUERY}

The URL part names which may be used for these conditions and actions are: Part Description HOST Full hostname. PATH URL substring beginning with the ﬁrst / after the hostname up to, but not including, the query string. PORT Port number. QUERY URL substring from the ?, signifying the beginning of the query parameters, until the end of the URL. Empty string if there were no quuery parameters. SCHEMEURL scheme in use (e.g. http and https). URL The complete URL. As another example, a remap rule might use the set-destination operator to change just the hostname via: cond%{HEADER:X-Mobile}="foo" set-destination HOST foo.mobile.bar.com [L]

Requests vs. Responses

Both HTTP requests and responses have headers, a good number of them with the same names. When writing a rule against, for example, the Connection or Via headers which are both valid in a request and a response, how can you tell which is which, and how do you indicate to Trafﬁc Server that the condition is speciﬁcally and exclusively for a request header or just for a response header? And how do you ensure that a header rewrite occurs against a request before it is proxied?

Hook Conditions

In addition to the conditions already described above, there are a set of special hook conditions. Only one of these conditions may be speciﬁed per ruleset, and they must be the ﬁrst condition listed. Which hook condition is used determines the context in which the ruleset is evaluated, and whether the other conditions will default to operating on the client request headers or the origin response (or cache response) headers. Hook conditions are written just like the other conditions, except that none of them take any operands: cond%{}

Because hook conditions must be the ﬁrst condition in a ruleset, the use of one forces the beginning of a new ruleset.

80 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

READ_RESPONSE_HDR_HOOK

Rulesets evaluated within this context will process only once the origin server response (or cached response) has been read, but prior to Trafﬁc Server sending that response to the client. This is the default hook condition for all globally-conﬁgured rulesets.

READ_REQUEST_HDR_HOOK

Evaluates as soon as the client request has been read, but prior to any further processing (including contacting origin servers or fetching objects from cache). Conditions and operators which adapt to matching or manipulating request or response entities (e.g. headers) depending on their context will all operate on the request variants when using this hook, as there is no response data yet.

READ_REQUEST_PRE_REMAP_HOOK

For ruleset conﬁgurations provided via remap.config, this forces their evaluation as soon as the request has been read, but prior to the remapping. For all context-adapting conditions and operators, matching will occur against the request, as there is no response data available yet.

REMAP_PSEUDO_HOOK

This is the default hook condition for all rulesets configured via remapping rules in remap.config. Functionally equivalent to READ_RESPONSE_HDR_HOOK in that rulesets will evaluate after responses from origin servers have been received (or the object has been retrieved from cache), but prior to sending the client response. What sets this hook context apart is that in configuration files shared by both the global plugin.config and individual remapping entries in remap.config, this hook condition will force the subsequent ruleset(s) to be valid only for remapped transactions.

SEND_REQUEST_HDR_HOOK

Forces evaluation of the ruleset just prior to contacting origin servers (or fetching the object from cache), but after any remapping may have occurred.

SEND_RESPONSE_HDR_HOOK

Evaluates rulesets just prior to sending the client response, but after any cache updates may have been performed. This hook context provides a means to modify aspects of the response sent to a client, while still caching the original versions of those attributes delivered by the origin server.

Affected Conditions

The following conditions are affected by the hook context in which they are evaluated and will adjust using request or response entities automatically: • HEADER • METHOD

3.7. Plugins 81 Apache-Trafﬁcserver-Server Documentation, latest

• PATH • QUERY • URL

Affected Operators

The following operators are affected by the hook context in which they are evaluated and will adjust modifying request or response entities automatically: • add-header • rm-header • set-header

Caveats

Repeated Headers

Some headers may appear more than once in a request or a response. When this occurs, all values will be collapsed into a single comma-delimited string before the conditions see them. This avoids the problem of determining which header instance out of several a condition’s rule will be applied to, but it may introduce unexpected behavior in your operands. For example, let us assume an origin response includes a header named X-Foo which speciﬁes a keyword of some sort. This header may appear zero or more times and we wish to construct a HEADER condition that can handle this.

Condition A

This condition will match using a direct equality operand:

cond%{HEADER:X-Foo}=bar

Condition B

This condition will match using an unanchored regular expression:

cond%{HEADER:X-Foo}/bar/

Sample Headers

Both conditions A and B will match this response:

HTTP/1.1 200OK Date: Mon, 08 Feb 2016 18:11:30 GMT Content-Length: 1234 Content-Type: text/html X-Foo: bar

82 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Only condition B will match this response:

HTTP/1.1 200OK Date: Mon, 08 Feb 2016 18:11:30 GMT Content-Length: 1234 Content-Type: text/html X-Foo: bar X-Foo: baz

That is because the HEADER condition will see the value of X-Foo as bar,baz. Condition B will still match this because the regular expression, being unanchored, ﬁnds the substring bar. But condition A fails, as it is expecting the value to be the exact string bar, nothing more and nothing less.

Examples

Remove Origin Authentication Headers

The following ruleset removes any authentication headers from the origin response before caching it or returning it to the client. This is accomplished by setting the hook context and then removing the cookie and basic authentication headers.:

cond%{READ_RESPONSE_HDR_HOOK} rm-header Set-Cookie rm-header WWW-Authenticate

Count Teapots

Maintains a counter statistic, which is incremented every time an origin server has decided to be funny by returning HTTP 418:

cond%{STATUS}=418 counter plugin.header_rewrite.teapots

Normalize Statuses

For client-facing purposes only (because we set the hook context to just prior to delivering the response back to the client, but after all processing and possible cache updates have occurred), replaces all 4xx HTTP status codes from the origin server with 404:

cond%{SEND_RESPONSE_HDR_HOOK} cond%{STATUS}>399 cond%{STATUS}<500 set-status 404

Remove Cache Control to Origins

Removes cache control related headers from requests before sending them to an origin server:

cond%{SEND_REQUEST_HDR_HOOK} rm-header Cache-Control rm-header Pragma

3.7. Plugins 83 Apache-Trafﬁcserver-Server Documentation, latest

Enable Debugging Per-Request

Turns on Trafﬁc Server debugging statements for a transaction, but only when a special header is present in the client request: cond%{READ_REQUEST_HDR_HOOK} cond%{CLIENT-HEADER:X-Debug}=supersekret set-debug

Remove Internal Headers

Removes special internal/development headers from origin responses, unless the client request included a special debug header: cond%{CLIENT-HEADER:X-Debug}=keep [NOT] rm-header X-Debug-Foo rm-header X-Debug-Bar

Return Original Method in Response Header

This rule copies the original HTTP method that was used by the client into a custom response header: cond%{SEND_RESPONSE_HDR_HOOK} set-header X-Original-Method%{METHOD}

Useless Example From Purpose

Even that useless example from Purpose in the beginning of this document is possible to accomplish: cond%{INCOMING-PORT}=8090 cond%{METHOD}=HEAD cond%{CLIENT-HEADER:Accept-Language}/es-py/ [NOT] cond%{STATUS}=304 [OR] cond%{RANDOM:500}>290 set-status 403

Add Cache Control Headers Based on Origin Path

This rule adds cache control headers to CDN responses based matching the origin path. One provides a max age and the other provides a “no-cache” statement to two different ﬁle paths.: cond%{SEND_RESPONSE_HDR_HOOK} cond%{PATH}/examplepath1/ add-header Cache-Control"max-age=3600" [L] cond%{SEND_RESPONSE_HDR_HOOK} cond%{PATH}/examplepath2/examplepath3/. */ add-header Cache-Control"no-cache" [L]

84 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Health Checks Plugin

This is a simple plugin, to provide basic (but configurable) health checks. This is a server intercept plugin, and it takes one single configuration option in plugin.config, the configuration file name.

Conﬁguration

To enable the healthchecks plugin, insert the following line in plugin.config:

healthchecks.so

The required may reference either an absolute or relative path to the file containing the healthcheck configuration. This configuration may contain one or more lines of the format:

: The URI-path can not be “/” only.

: This conﬁguration is not reloadable.

The content of the file specified in the file-path, if any, is sent as the body of the response. The existence of the file is sufficient to get an “OK” status. Performance wise, everything is served out of memory, and it only stats / opens files as necessary. However, the content of the status file is limited to 16KB, so this is not a generic static file serving plugin.

Example

This line would define a health check link available at http://www.example.com/__hc that would check if the file / var/run/ts-alive existed on the server. If the file exists, a response is built with the contents of the ts-alive file, a mime type of text/plain and a status code of 200. If the file does not exist, a 403 response is sent:

/__hc/var/run/ts-alive text/plain 200 403

Regex Remap Plugin

This allows you to configure mapping rules based on regular expressions. This is similar to what you can accomplish using mod_rewrite in Apache httpd, but obviously not as flexible or sophisticated (yet). To use this plugin, configure a remap.config rule like

map http://a.com http://b.com @plugin=regex_remap.so @pparam=maps.reg

The file name parameter is always required. Unless an absolute path is specified, the file name is assumed to be a path relative to the Traffic Server configuration directory. The regular expressions listed in the configuration file are evaluated sequentially. When a regular expression is pos- itively matched against a request URL, evaluation is stopped and the rewrite rule is applied. If none of the regular expressions are a match, the default destination URL is applied (http://b.com in the example above).

3.7. Plugins 85 Apache-Trafﬁcserver-Server Documentation, latest

An optional argument (@pparam) with the string “profile” will enable proﬁling of this regex remap rule, e.g.

... @pparam=maps.reg @pparam=profile

Profiling is very low overhead, and the information is dumped to traffic.out, located in the log directory. This information is useful to optimize the order of your regular expression, such that the most common matches appears early in the file. In order to force a profile dump, you can do

$ sudo touch remap.config $ sudo traffic_ctl config reload

By default, only the path and query string of the URL are provided for the regular expressions to match. The following optional parameters can be used to modify the plugin instance behavior

@pparam=[no-]method [default: off] @pparam=[no-]query-string [default: on] @pparam=[no-]matrix-parameters [default: off]

If you wish to match on the HTTP method used (e.g. “GET”), you must use the option @pparam=method. e.g.

... @pparam=maps.reg @pparam=method

With this enabled, the string that you will need to match will look like

GET/path?query=bar

The methods are always all upper-case, and always followed by one single space. There is no space between the method and the rest of the URL (or URI path). By default, the query string is part of the string that is matched again, to turn this off use the option ‘no-query-string’, e.g.

... @pparam=maps.reg @pparam=no-query-string

Finally, you can also include the matrix parameters in the string, using the option ‘matrix-parameters’, e.g.

... @pparam=maps.reg @pparam=matrix-parameters

A typical regex would look like

^/(ogre.*)/more http://www.ogre.com/$h/$0/$1

The regular expression must not contain any white spaces! When the regular expression is matched, only the URL path + query string is matched (without any of the optional conﬁguration options). The path will always start with a “/”. Various substitution strings are allowed on the right hand side during evaluation

$0 - The entire matched string $1-9 - Regular expression groups ($1 first group etc.) $h - The original host header from the request $f - The host as used in the "from" portion of the remap rule $t - The host as used in the "to" portion of the remap rule $p - The original port number $s - The scheme (e.g. http) of the request $P - The entire path of the request $q - The query part of the request $r - The path parameters of the request (not implemented yet)

86 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

$c - The cookie string from the request $i - The client IP for this request

: The $0 substitution expands to the characters that were matched by the regular expression, not to the entire string that the regular expression was matched against.

You can also provide options, similar to how you conﬁgure your remap.conﬁg. The following options are available

@status= - Force the response code to @active_timeout= - Active timeout (in ms) @no_activity_timeout= - No activity timeout (in ms) @connect_timeout= - Connect timeouts (in ms) @dns_timeout= - Connect timeouts (in ms)

@overridable-config= - see :ref:`Overridable Configurations

˓→config>`

@caseless - Make regular expressions case insensitive @lowercase_substitutions - Turn on (enable) lower case substitutions

This can be useful to force a particular response for some URLs, e.g.

^/(ogre.*)/bad http://www.examle.com/ @status=404

Or, to force a 302 redirect

^/oldurl/(.*)$ http://news.example.com/new/$1 @status=302

Setting the status to 301 or 302 will force the new URL to be used as a redirect (Location:).

Stats Over HTTP Plugin

This plugin implements an HTTP interface to all Trafﬁc Server statistics. The metrics returned are in a JSON format, for easy processing. This plugin is now part of the standard ATS build process, and should be available after install.

Enabling Stats Over HTTP

To enable this plugin, add to the plugin.config ﬁle: stats_over_http.so

After starting Trafﬁc Server, the JSON metrics are now available on the default URL: http://host:port/_stats where host and port is the hostname/IP and port number of the server.

Plugin Options

--integer-counters

3.7. Plugins 87 Apache-Trafﬁcserver-Server Documentation, latest

This option causes the plugin to emit ﬂoating point and integral metric values as JSON numbers, rather then JSON strings. This can cause interoperability problems since integer metrics have a 64-bit unsigned range. --wrap-counters This option wraps 64-bit unsigned values to the 64-bit signed range. This aids interoperability with Java, since prior to the Java SE 8 release, Java did not have a 64-bit unsigned type. You can optionally modify the path to use, and this is highly recommended in a public facing server. For example:

stats_over_http.so 81c075bc0cca1435ea899ba4ad72766b

and the URL would then be e.g.:

https://host:port/81c075bc0cca1435ea899ba4ad72766b

This is weak security at best, since the secret could possibly leak if you are careless and send it over clear text.

TCPInfo Plugin

This global plugin logs TCP metrics at various points in the HTTP processing pipeline. The TCP information is retrieved by the getsockopt(2) function using the TCP_INFO option. This is only supported on systems that support the TCP_INFO option, currently Linux and BSD.

Plugin Options

The following options may be specified in plugin.config: --hooks=NAMELIST This option specifies when TCP information should be logged. The argument is a comma-separated list of the event names listed below. TCP information will be sampled and logged each time the specified set of events occurs. Event Name Triggered when send_resp_hdr The server begins sending an HTTP response. ssn_close The TCP connection closes. ssn_start A new TCP connection is accepted. txn_close A HTTP transaction is completed. txn_start A HTTP transaction is initiated. --log-file=NAME This specifies the base name of the file where TCP information should be logged. If this option is not specified, the name tcpinfo is used. Traffic Server will automatically append the .log suffix. --log-level=LEVEL The log level can be either 1 to log only the round trip time estimate, or 2 to log the complete set of TCP information. The following fields are logged when the log level is 1: Field Name Description timestamp Event timestamp event Event name (one of the names listed above) client Client IP address server Server IP address rtt Estimated round trip time The following fields are logged when the log level is 2:

88 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Field Name Description timestamp Event timestamp event Event name (one of the names listed above) client Client IP address server Server IP address rtt Estimated round trip time rttvar last_sent last_recv snd_cwnd snd_ssthresh rcv_ssthresh unacked sacked lost retrans fackets --sample-rate=COUNT This is the number of times per 1000 requests that the data will be logged. A pseudo-random number generator is used to determine if a request will be logged. The default value is 1000 and this option is not required to be in the conﬁguration ﬁle. To achieve a log rate of 1% you would set this value to 10.

Examples:

This example logs the simple TCP information to tcp-metrics.log at the start of a TCP connection and once for each HTTP transaction thereafter: tcpinfo.so--log-file=tcp-metrics--log-level=1--hooks=ssn_start,txn_start

The ﬁle tcp-metrics.log will contain the following log format: timestamp event client server rtt 20140414.17h40m14s ssn_start 127.0.0.1 127.0.0.1 4000 20140414.17h40m14s txn_start 127.0.0.1 127.0.0.1 4000 20140414.17h40m16s ssn_start 127.0.0.1 127.0.0.1 4000 20140414.17h40m16s txn_start 127.0.0.1 127.0.0.1 4000 20140414.17h40m16s ssn_start 127.0.0.1 127.0.0.1 4000 20140414.17h40m16s txn_start 127.0.0.1 127.0.0.1 4000

Cache URL Modify the cache key used for requests by applying a regular expression to the URL. Configuration Remap Override configuration directives on a per-rule basis. GZip Compress or deflate cache responses. Header Rewrite Modify requests and responses based on incoming and outgoing headers and other transaction attributes. Health Checks Define service health check links. Regex Remap Configure remapping rules using regular expressions. Stats over HTTP Provide an HTTP interface to all Traffic Server statistics. TCPInfo Log TCP metrics at various points of the HTTP processing pipeline.

3.7. Plugins 89 Apache-Trafﬁcserver-Server Documentation, latest

Experimental plugins

Plugins that are considered experimental are located in the plugins/experimental directory of the Trafﬁc Server source tree. Experimental plugins can be compiled by passing the –enable-experimental-plugins option to conﬁgure:

$ autoconf -i $ ./configure --enable-experimental-plugins $ make

AuthProxy Plugin

There are many ways of authorizing an HTTP request. Often, this requires making IPC calls to some internal infrastructure. AuthProxy is a plugin that takes care of the Traffic Server end of authorizing a request and delegates the authorization decision to an external HTTP service. This plugin can be used as either a global plugin or a remap plugin. Note that Traffic Server optimizes latency by skipping the DNS lookup state if a document is found in the cache. This will have the effect of serving the document without consulting the AuthProxy plugin. you can disable this behavior by setting :ts:cv:‘proxy.config.http.doc_in_cache_skip_dns‘ to 0 in records.config. Note that the authorization request will need to match a remap rule (which, as a standalone remap rule, does not need to call the AuthProxy plugin). If a second remap rule is required, by default, the authorization request will not have the same Host header as the request from the client. It could be added using the header_rewrite plugin (set-header Host “pristine_host.example.com”).

Plugin Options

--auth-transform=TYPE This option speciﬁes how to route the incoming request to the authorization service. The transform type may be head or redirect. If the transform type is head, then the incoming request is transformed to a HEAD request and is sent to the same destination. If the response is 200 OK, the incoming request is allowed to proceed. If the transform type is range, then the incoming request is transformed to a Range request asking for 0 bytes. Other than that, the behavior is identical to the head option above. This type of Range request is useful when the upstream destination is a cache, and it’s not able to cache HEAD requests. If the transform type is redirect then the incoming request is sent to the authorization service designated by the –auth-host and –auth-port parameters. If the response is 200 OK, the incoming request is allowed to proceed. When the authorization service responds with a status other than 200 OK, that response is returned to the client as the response to the incoming request. This allows mechanisms such as HTTP basic authentication to work correctly. Note that the body of the authorization service response is not returned to the client. --auth-host=HOST The name or address of the authorization host. This is only used by the redirect transform. --auth-port=PORT The TCP port of the authorization host. This is only used by the redirect transform.

90 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

--force-cacheability If this options is set, the plugin will allow Traffic Server to cache the result of authorized requests. In the normal case, requests with authorization headers are nor cacheable, but this flag allows that by setting the :ts:cv:‘proxy.config.http.cache.ignore_authentication‘ option on the request.

Examples

In this example, the authentication is performed by converting the incoming HTTP request to a HEAD request and sending that to the origin server origin.internal.com:

map http://cache.example.com http://origin.internal.com/\ @plugin=authproxy.so @pparam=--auth-transform=head map http://origin.internal.com http://origin.internal.com/

In this example, the request is directed to a local authentication server that authorizes the request based on internal policy rules:

map http://cache.example.com http://origin.internal.com/\ @plugin=authproxy.so @pparam=--auth-transform=redirect @pparam=--auth-host=127.0.0.

˓→1 @pparam=--auth-port=9000

map http://origin.internal.com/ http://origin.internal.com/\ @plugin=authproxy.so @pparam=--auth-transform=redirect @pparam=--auth-host=127.0.0.

˓→1 @pparam=--auth-port=9000

AWS S3 Authentication plugin

This is a plugin for Apache Trafﬁc Server that provides support for the Amazon S3 authentication features. This is useful if you for example want to use S3 as your origin server, yet want to avoid direct user access to the content.

Using the plugin

There are three conﬁguration options for this plugin:

--access_key --secret_key --virtual_host --config

Using the ﬁrst two in a remap rule would be e.g.:

... @plugin=s3_auth @pparam=--access_key @pparam=my-key \ @pparam=--secret_key @pparam=my-secret \ @pparam=--virtual_host

Alternatively, you can store the access key and secret in an external configuration file, and point the remap rule(s) to it: ... @plugin=s3_auth @pparam=–config @pparam=s3.config Where s3.config would look like:

3.7. Plugins 91 Apache-Trafﬁcserver-Server Documentation, latest

# AWS S3 authentication access_key=my-key secret_key=my-secret virtual_host=yes

For more details on the S3 auth, see: http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html

ToDo

This is a pretty barebone start for the S3 services, it is missing a number of features: • It does not do UTF8 encoding (as required) • It only implements the v2 authentication mechanism. For details on v4, see http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html • It does not deal with canonicalization of AMZ headers. • It does not handle POST requests (but do we need to ?) • It does not incorporate query parameters. Contributions to any of these would be appreciated.

Background Fetch Plugin

This is a plugin for Apache Traffic Server that allows you to proactively fetch content from Origin in a way that it will fill the object into cache. This is particularly useful when all (or most) of your client requests are of the byte-Range type. The underlying problem being that Traffic Server is not able to cache request / responses with byte ranges.

Using the plugin

This plugin functions as either a global or per remap plugin, and it takes an optional argument for specifying a config file with inclusion or exclusion criteria. The config file can be specified both via an absolute path or via a relative path to the install dir To activate the plugin in global mode, in plugin.config, simply add: background_fetch.so--config

To activate the plugin in per remap mode, in remap.config, simply append the below to the speciﬁc remap line:

@plugin=background_fetch.so @pparam=

Functionality

Examining the responses from origin, we decide to trigger a background fetch of the original (Client) request under these conditions: • The request is a GET request (we only support these right now) • The response is a 206 response

92 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

• The original client request, and the Origin server response, is clearly indicating that the response is cacheable. This uses the new API c:func:TSHttpTxnIsCacheable(), which also implies honoring current Traffic Server configurations. Once deemed a good candidate to performance a background fetch, we’ll replay the original client request through the Traffic Server proxy again, except this time eliminating the Range header. This is transparent to the original client request, which continues as normal. Only one background fetch per URL is ever performed, making sure we do not accidentally put pressure on the origin servers. The plugin now supports a config file that can specify exclusion or inclusion of background fetch based on any arbitrary header or client-ip:

background_fetch.so--config

The contents of the conﬁg-ﬁle could be as below:

include User-Agent ABCDEF exclude User-Agent * exclude Content-Type text exclude X-Foo-Bar text exclude Content-Length<1000

The plugin also now supports per remap activation. To activate the plugin for a given remap, add the below on the remap line:

@plugin=background_fetch.so @pparam=

Future additions

• Limiting the background fetches to content of certain sizes

Balancer Plugin

The balancer balances requests across multiple origin servers. To use this plugin, conﬁgure it in a remap.config rule, specifying a balancing policy and a set of origin servers. For example:

map http://foo.com http://foo.com \ @plugin=balancer.so @pparam=--policy=hash,url @pparam=one.bar.com @pparam=two.bar.

˓→com

The replacement URL in the mapping rule is not used. The argument to the --policy option is a comma- separated list of keywords. The ﬁrst keyword is the name of a balancing policy. The subsequent keywords are used to reﬁne the requested policy. The remaining plugin arguments are balancer targets. Typically, these will be the host names of origin servers that requests should be balanced across. The target name may contain a colon-separated port number.

Hash Balancing Policy

The hash balancing policy performs a consistent hash across the set of origins. This minimizes the number of hash entries that must be moved when the set of origin servers changes. An optional list of hash fields follows the hash keyword. Each specified hash field is hashed to select an outbound origin server.

3.7. Plugins 93 Apache-Trafﬁcserver-Server Documentation, latest

The following fields can be supplied to the hash: key The request cache key. Note that the cache key will only be set if you have already chained a plugin that sets a custom cache key. url The request URL. This is the default hash field that is used if no other fields are specified. srcaddr The source IP address of the request. dstaddr The destination IP address of the request.

Round Robin Balancing Policy

The roundrobin balancing policy simply allocates requests to origin servers in order. Over time, the number of requests received by each origin should be approximately the same.

Health Checking

The balancer plugin does not check the health of the origin servers, however the plugin is fully reloadable so health checking is usualy simple to implement. Most production environments already have mechanisms to check service health. It is recommended that you write a simple script to monitor this information and rewrite remap.config when appropriate. Running traffic_ctl config reload will reload the balancer plugin with the new set of origin servers.

Buffer Upload Plugin

The Buffer Upload plugin offers the following features

Installation

Conﬁguration can be explicitly speciﬁed as a parameter in plugin.config buffer_upload.so/FOOBAR/upload.conf

Memory buffering (buffer the entire POST data in IOBuffer before connecting to OS)

Memory buffer size is configured with “mem_buffer_size” in config file. Default and minimum value is 32K. You can increase it in the config file. If the size of a request is larger than the “mem_buffer_size” value specifiied in the config file, then the upload proxy feature will be disabled for this particular request

Disk buffering (buffer the entire POST data on disk before connecting to OS)

1. Disk async IO is used. AIO api call only involves certain amount of threads. The number of threads is configurable in plugin’s config file (default is 4) 2. Directories and files are generated on disk . Base directory is /FOOBAR/var/buffer_upload_tmp/ (configurable in config file). Number of subdirectories is 64 (configurable in config file). Filename are randomly generated. Files will be removed when the entire data have been sent out to OS . At startup time, dangling files are removed (left on disk due to transaction interruption or traffic server crash) 3. Default chunk size when reading from disk is 16K, configurable in config file

94 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Trigger POST buffering on certain URLs

1. Certain URLs will be provided in a plain text file (one URL each line) 2. Specify filename in config file by “url_list_file” 3. max length of each URL is 4096 (configurable in config file) 4. use exact match, don’t support regex for now

Other Features

1. Default buffering mode is disk aio buffering mode. To turn off disk buffering, add a “use_disk_buffer 0” line in conﬁg ﬁle 2. All request headers inlcuding cookies plus the entire POST data will be buffered (either in memory or on disk)

Configuration File sample config file use_disk_buffer1 convert_url1 chunk_size 1024 url_list_file/tmp/url_list.conf max_url_length 10000 base_dir/tmp/test1 subdir_num 100 thread_num 10 mem_buffer_size 40000

Cache Key Manipulation Plugin

Description

This plugin allows some common cache key manipulations based on various HTTP request components. It can • sort query parameters to prevent query parameters reordereding from being a cache miss • ignore specific query parameters from the cache key by name or regular expression • ignore all query parameters from the cache key • only use specific query parameters in the cache key by name or regular expression • include headers or cookies by name • capture values from the User-Agent header. • classify request using User-Agent and a list of regular expressions • capture and replace strings from the URI and include them in the cache key • do more - please find more examples below.

3.7. Plugins 95 Apache-Trafﬁcserver-Server Documentation, latest

Cache key structure and related plugin parameters

hierarchical part query ------| Prefix | User-Agent | Headers | Cookies | Path | Query

˓→ | ------

• The cache key set by the cachekey plugin can be considered as devided into several sections. • Every section is manupulated separately by the related plugin parameters (more info in each section description below). • “User-Agent”, “Headers” and “Cookies” sections are optional and will be missing from the cache key if no related plugin parameters are used. • “Preﬁx”, “Path” and “Query” sections always have default values even if no related plugin parameters are used. • All cachekey plugin parameters are optional and if missing some of the cache key sections will be missing (the optional sections) or their values will be left to their defaults.

“Preﬁx” section

Optional components | ------(included in this order) | | --static-prefix | --capture-prefix | --capture-prefix-

˓→uri | | ------Default values if no | | /host/port

˓→ | optional components | ------configured |

• --static-prefix= (default: empty string) - if specified and not an empty string the will be added to the cache key. • --capture-prefix= (default: empty string) - if specified and not empty then strings are captured from host:port based on the and are added to the cache key. • --capture-prefix-uri= (default: empty string) - if specified and not empty then strings are captured from the entire URI based on the and are added to the cache key. • If any of the “Prefix” related plugin parameters are used together in the plugin configuration they are added to the cache key in the order shown in the diagram.

“User-Agent” section

Optional components | ------(included in this order) | | --ua-class | --ua-capture | | ------

96 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Default values if no | | (empty) | optional components | ------configured |

• User-Agent classification – --ua-whitelist=: (default: empty string) - loads a regex patterns list from a file , the patterns are matched against the User-Agent header and if matched is added it to the key. – --ua-blacklist=: (default: empty string) - loads a regex patterns list from a file , the patterns are matched against the User-Agent header and if not matched is added it to the key. – Multiple --ua-whitelist and --ua-blacklist can be used and the result will be defined by their order in the plugin configuration. • User-Agent regex capturing and replacement – --ua-capture= (default: empty string) - if specified and not empty then strings are captured from the User-Agent header based on (see below) and are added to the cache key. • If any User-Agent classigication and regex capturing and replacement plugin parameters are used together they are added to the cache key in the order shown in the diagram.

“Headers” section

Optional components | ------| | --include-headers | | ------Default values if no | | (empty) | optional components | ------configured |

• --include-headers (default: empty list) - comma separated list of headers to be added to the cache key. The list of headers deﬁned by --include-headers are always sorted before adding them to the cache key.

“Cookies” section

Optional components | ------| | --include-cookies | | ------Default values if no | | (empty) | optional components | ------configured |

• --include-cookies (default: empty list) - comma separated list of cookies to be added to the cache key. The list of cookies deﬁned by --include-cookies are always sorted before adding them to the cache key.

“Path” section

3.7. Plugins 97 Apache-Trafﬁcserver-Server Documentation, latest

Optional components | ------(included in this order) | | --path-capture-uri | --path-capture | | ------Default values if no | | URI path | optional components | ------configured |

• if no path related plugin parameters are used, the URI path string is included in the cache key. • --capture-path= (default: empty string) - if speciﬁed and not empty then strings are captured from URI path based on the and are added to the cache key. • --capture-path-uri= (default: empty string) - if speciﬁed and not empty then strings are captured from the entire URI based on the and are added to the cache key.

“Query” section

• If no query related plugin parameters are used, the query string is included in the cache key. • --exclude-params (default: empty list) - comma-separated list of query params to be black-listed in the cache key. If the list is empty then no black-list is applied (no query parameters will be excluded from the cache key). The exclude list overrides the include list. • --include-params (default: empty list) - comma-separated list of query params to be white-listed in the cache key. If the list is empty then no white-list is applied (all query parameters will be included in the cache key). • --include-match-params (default: empty list) - regular expression matching query parameter names which will be white-listed in the cache key. • --exclude-match-params (default: empty list) - regular expression matching query parameter names which will be black-listed in the cache key. • --remove-all-params (boolean:true|false, 0|1, yes|no, default: false) - if equals true then all query parameters are removed (the whole query string) and all other URI query parameter related settings (if used) will have no effect. • --sort-params (boolean:true|false, 0|1, yes|no, default: false) - if equals true then all query parameters are sorted in an increasing case-sensitive order All parameters are optional, and if not used, their default values are as mentioned below. Boolean values default to false and the rest default to an empty list. Examples of each parameter’s usage can be found below.

• can be in the following formats – - defines regex capturing groups, up to 10 captured strings based on will be added to the cache key. – /// - defines regex capturing groups, defines a pattern where the captured strings referenced with $0 ... $9 will be substituted and the result will be added to the cache key.

98 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Detailed examples and troubleshooting

| hierarchical part

˓→ query HTTP request | ------components | URI host and port HTTP headers and cookies URI

˓→path URI query | ------Sample 1 | /www.example.com/80/popular/Mozilla/5.0/H1:v1/H2:v2/C1=v1;C2=v2/path/

˓→to/data?a=1&b=2&c=3 Sample 2 | /nice_custom_prefix/popular/Mozilla/5.0/H1:v1/H2:v2/C1=v1;C2=v2/path/

˓→to/data?a=1&b=2&c=3 | ------Cache Key | host:port or UA-class UA-captures headers cookies

˓→path query components | custom prefix replacement

The following is an example of how the above sample keys were generated (Sample 1 and Sample 2). Trafﬁc server conﬁguration

$ cat etc/trafficserver/remap.config map http://www.example.com http://www.origin.com \ @plugin=cachekey.so \ @pparam=--ua-whitelist=popular:popular_agents.config \ @pparam=--ua-capture=(Mozilla\/[^\s]*).* \ @pparam=--include-headers=H1,H2 \ @pparam=--include-cookies=C1,C2 \ @pparam=--include-params=a,b,c \ @pparam=--sort-params=true

$ cat etc/trafficserver/popular_agents.config ^Mozilla.* ^Twitter.* ^Facebo.*

$ cat etc/trafficserver/plugin.config xdebug.so

HTTP request

$ curl 'http://www.example.com/path/to/data?c=3&a=1&b=2&x=1&y=2&z=3' \ -v -x 127.0.0.1:8080 -o /dev/null -s \ -H "H1: v1" \ -H "H2: v2" \ -H "Cookie: C1=v1; C2=v2" \ -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.

˓→14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A' \ -H 'X-Debug: X-Cache-Key' * About to connect() to proxy 127.0.0.1 port 8080 (#0) * Trying 127.0.0.1... connected * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0) > GET http://www.example.com/path/to/data?c=3&a=1&b=2&x=1&y=2&z=3 HTTP/1.1 > Host: www.example.com > Accept: */* > Proxy-Connection: Keep-Alive > H1: v1 > H2: v2

3.7. Plugins 99 Apache-Trafﬁcserver-Server Documentation, latest

> Cookie: C1=v1; C2=v2 > User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14

˓→(KHTML, like Gecko) Version/7.0.3 Safari/7046A194A > X-Debug: X-Cache-Key > < HTTP/1.1 200 OK < Server: ATS/6.1.0 < Date: Thu, 19 Nov 2015 23:17:58 GMT < Content-type: application/json < Age: 0 < Transfer-Encoding: chunked < Proxy-Connection: keep-alive < X-Cache-Key: /www.example.com/80/popular/Mozilla/5.0/H1:v1/H2:v2/C1=v1;C2=v2/path/

˓→to/data?a=1&b=2&c=3 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact * Closing connection #0

The response header X-Cache-Key header contains the cache key:

/www.example.com/80/popular/Mozilla/5.0/H1:v1/H2:v2/C1=v1;C2=v2/path/to/data?a=1&b=2&

˓→c=3

The xdebug.so plugin and X-Debug request header are used just to demonstrate basic cache key troubleshooting. If we add --static-prefix=nice_custom_prefix to the remap rule then the cache key would look like the following:

/nice_custom_prefix/popular/Mozilla/5.0/H1:v1/H2:v2/C1=v1;C2=v2/path/to/data?a=1&b=2&

˓→c=3

Usage examples

URI query parameters

Ignore the query string (all query parameters)

The following added to the remap rule will ignore the query, removing it from the cache key.

@plugin=cachekey.so @pparam=--remove-all-params=true

Cache key normalization by sorting the query parameters

The following will normalize the cache key by sorting the query parameters.

@plugin=cachekey.so @pparam=--sort-params=true

If the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1 the cache key will use a=1&b=2&c=1&k=1&u=1&x=1&y=1

100 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Ignore (exclude) certain query parameters

The following will make sure query parameters a and b will not be used when constructing the cache key.

@plugin=cachekey.so @pparam=--exclude-params=a,b

If the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1 the cache key will use c=1&x=1&k=1&u=1&y=1

Ignore (exclude) certain query parameters from the cache key by using regular expression (PCRE)

The following will make sure query parameters a and b will not be used when constructing the cache key.

@plugin=cachekey.so @pparam=--exclude-match-params=(a|b)

If the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1 the cache key will use c=1&x=1&k=1&u=1&y=1

Include only certain query parameters

The following will make sure only query parameters a and c will be used when constructing the cache key and the rest will be ignored.

@plugin=cachekey.so @pparam=--include-params=a,c

If the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1 the cache key will use c=1&a=1

Include only certain query parameters by using regular expression (PCRE)

The following will make sure only query parameters a and c will be used when constructing the cache key and the rest will be ignored.

@plugin=cachekey.so @pparam=--include-match-params=(a|c)

If the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1 the cache key will use c=1&a=1

White-list + black-list certain parameters using multiple parameters in the same remap rule.

If the plugin is used with the following plugin parameters in the remap rule:

@plugin=cachekey.so \ @pparam=--exclude-params=x \ @pparam=--exclude-params=y \ @pparam=--exclude-params=z \ @pparam=--include-params=y,c \ @pparam=--include-params=x,b and if the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1 the cache key will use c=1&b=1

3.7. Plugins 101 Apache-Trafﬁcserver-Server Documentation, latest

White-list + black-list certain parameters using multiple parameters in the same remap rule and regular expressions (PCRE).

If the plugin is used with the following plugin parameters in the remap rule:

@plugin=cachekey.so \ @pparam=--exclude-match-params=x \ @pparam=--exclude-match-params=y \ @pparam=--exclude-match-params=z \ @pparam=--include-match-params=(y|c) \ @pparam=--include-match-params=(x|b)

and if the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1 the cache key will use c=1&b=1

Mixing –include-params, –exclude-params, –include-match-param and –exclude-match-param

If the plugin is used with the following plugin parameters in the remap rule:

@plugin=cachekey.so \ @pparam=--exclude-params=x \ @pparam=--exclude-match-params=y \ @pparam=--exclude-match-params=z \ @pparam=--include-params=y,c \ @pparam=--include-match-params=(x|b)

and if the URI has the following query string c=1&a=1&b=2&x=1&k=1&u=1&y=1 the cache key will use c=1&b=1

HTTP Headers

Include certain headers in the cache key

The following headers HeaderA and HeaderB will be used when constructing the cache key and the rest will be ignored.

@plugin=cachekey.so @pparam=--include-headers=HeaderA,HeaderB

HTTP Cookies

Include certain cookies in the cache key

The following headers CookieA and CookieB will be used when constructing the cache key and the rest will be ignored.

@plugin=cachekey.so @pparam=--include-headers=CookieA,CookieB

Preﬁx (host, port, capture and replace from URI)

102 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

Replacing host:port with a static cache key preﬁx

If the plugin is used with the following plugin parameter in the remap rule.

@plugin=cachekey.so @pparam=--static-prefix=static_prefix

the cache key will be preﬁxed with /static_prefix instead of host:port when --static-prefix is not used.

Capturing from the host:port and adding it to the preﬁx section

If the plugin is used with the following plugin parameter in the remap rule.

@plugin=cachekey.so \ @pparam=--capture-prefix=(test_prefix).*:([^\s\/$]*)

the cache key will be preﬁxed with /test_prefix/80 instead of test_prefix_371.example.com:80 when --capture-prefix is not used.

Capturing from the entire URI and adding it to the preﬁx section

If the plugin is used with the following plugin parameter in the remap rule.

@plugin=cachekey.so \ @pparam=--capture-prefix-uri=/(test_prefix).*:.*(object).*$/$1_$2/

and if the request URI is the following

http://test_prefix_123.example.com/path/to/object?a=1&b=2&c=3

the the cache key will be preﬁxed with /test_prefix_object instead of test_prefix_123.example. com:80 when --capture-prefix-uri is not used.

Combining prefix plugin parameters, i.e. –static-prefix and –capture-prefix

If the plugin is used with the following plugin parameters in the remap rule.

@plugin=cachekey.so \ @pparam=--capture-prefix=(test_prefix).*:([^\s\/$]*)\ @pparam=--static-prefix=static_prefix

the cache key will be preﬁxed with /static_prefix/test_prefix/80 instead of test_prefix_371. example.com:80 when neither --capture-prefix nor --static-prefix are used.

Path, capture and replace from the path or entire URI

Capture and replace groups from path for the “Path” section

If the plugin is used with the following plugin parameter in the remap rule.

3.7. Plugins 103 Apache-Trafﬁcserver-Server Documentation, latest

@plugin=cachekey.so \ @pparam=--capture-path=/.*(object).*/const_path_$1/ and the request URI is the following http://test_path_123.example.com/path/to/object?a=1&b=2&c=3 then the cache key will have /const_path_object in the path section of the cache key instead of /path/to/ object when neither --capture-path nor --capture-path-uri are used.

Capture and replace groups from whole URI for the “Path” section

If the plugin is used with the following plugin parameter in the remap rule.

@plugin=cachekey.so \ @pparam=--capture-path-uri=/(test_path).*(object).*/$1_$2/ and the request URI is the following http://test_path_123.example.com/path/to/object?a=1&b=2&c=3 the the cache key will have /test_path_object in the path section of the cache key instead of /path/to/ object when neither --capture-path nor --capture-path-uri are used.

Combining path plugin parameters –capture-path and –capture-path-uri

If the plugin is used with the following plugin parameters in the remap rule.

@plugin=cachekey.so \ @pparam=--capture-path=/.*(object).*/const_path_$1/ \ @pparam=--capture-path-uri=/(test_path).*(object).*/$1_$2/ and the request URI is the following http://test_path_123.example.com/path/to/object?a=1&b=2&c=3 the the cache key will have /test_path_object/const_path_object in the path section of the cache key instead of /path/to/object when neither --capture-path nor --capture-path-uri are used.

User-Agent capturing, replacement and classiﬁcation

Let us say we have a request with User-Agent header:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A

Capture PCRE groups from User-Agent header

If the plugin is used with the following plugin parameter:

104 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest

@plugin=cachekey.so \ @pparam=--ua-capture=(Mozilla\/[^\s]*).*(AppleWebKit\/[^\s]*) then Mozilla/5.0 and AppleWebKit/537.75.14 will be used when constructing the key.

Capture and replace groups from User-Agent header

If the plugin is used with the following plugin parameter:

@plugin=cachekey.so \ @pparam=--ua-capture=/(Mozilla\/[^\s]*).*(AppleWebKit\/[^\s]*)/$1_$2/ then Mozilla/5.0_AppleWebKit/537.75.14 will be used when constructing the key.

User-Agent white-list classiﬁer

If the plugin is used with the following plugin parameter:

@plugin=cachekey.so \ @pparam=--ua-whitelist=browser:browser_agents.config and if browser_agents.config contains:

^Mozilla.* ^Twitter.* ^Facebo.* then browser will be used when constructing the key.

User-Agent black-list classiﬁer

If the plugin is used with the following plugin parameter:

@plugin=cachekey.so \ @pparam=--ua-blacklist=browser:tool_agents.config and if tool_agents.config contains:

^PHP.* ^Python.* ^curl.* then browser will be used when constructing the key.

Cache Promote Plugin

The cache_promote plugin provides a means to control when an object should be allowed to enter the cache. This is orthogonal from normal Cache-Control directives, providing a different set of policies to apply. The typical use case for this plugin is when you have a very large data set, where you want to avoid churning the ATS cache for the long tail content. All conﬁguration is done via remap.config, and the following options are available:

3.7. Plugins 105 Apache-Trafﬁcserver-Server Documentation, latest

--policy The promotion policy. lru and chance are supported --sample The sampling rate for the request to be considered For the lru plugin, the following options are also available: --hits The minimum number of hits before promotion --buckets The size (entries) of the LRU These two options combined with your usage patterns will control how likely a URL is to become promoted to enter the cache.

Examples

map http://cdn.example.com/ http://some-server.example.com @plugin=cache_promote.so @pparam=–policy=chance –sample=10% map http://cdn.example.com/ http://some-server.example.com @plugin=cache_promote.so @pparam=–policy=lru @pparam=–hits=10 @pparam=–buckets=10000 Note that the –sample option is available for all policies, and when used to reduce pressure on the under heavy load.

Combo Handler Plugin

This plugin provides an intelligent way to combine multiple URLs into a single URL, and have Apache Trafﬁc Server combine the components into one response. This is useful for example to create URLs that combine multiple CSS or Javascript ﬁles into one.

Installation

This plugin is only built if the conﬁgure option

--enable-experimental-plugins is given at build time. Note that this plugin is built and installed in combination with the ESI module, since they share common code.

Conﬁguration

The arguments in the plugin.config line in order represent 1. The path that should triggers combo handler (defaults to “admin/v1/combo”) 2. The name of the key used for signature veriﬁcation (disabled by default) A “-” can be supplied as a value for any of these arguments to request default value be applied. Also, just like the original combohandler, this plugin generates URLs of the form http://localhost/

/ .  here defaults to l unless speciﬁed by the ﬁle path in the query parameter using a colon. For example: http://combo.com/admin/v1/combo?filepath1&dir1:filepath2&filepath3
Will result in these three pages being fetched:
106 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 http://localhost/l/filepath1 http://localhost/dir1/filepath2 http://localhost/l/filepath3
Remap rules have to be speciﬁed to map the above URLs to desired content servers. The plugin also supports a preﬁx parameter. Common parts of successive ﬁle paths can be extracted and speciﬁed separately using a ‘p’ query parameter. Successive ﬁle path parameters are appended to this preﬁx to create complete ﬁle paths. The preﬁx will remain active until changed or cleared (set to an empty string). For example, the query
"/file1&p=/path1/&file2&file3&p=&/file4&p=/dir:path2/&file5&file6" results in these ﬁle paths being “reconstructed”:
/file1 /path1/file2 /path1/file3 /file4 /dir:path2/file5 /dir:path2/file6
Epic Plugin
The Epic plugin emits Trafﬁc Server metrics in a format that is consumed by the Epic Network Monitoring System. It is a global plugin that is installed by adding it to the plugin.config ﬁle.
Plugin Options
--directory=DIR Specify the directory the plugin will write sample ﬁles to. The default is /usr/ local/epic/cache/eapi. --period=SECS Specify the sample period in seconds. The default is to write samples every 30 seconds.
Caveats
The Trafﬁc Server metrics system does not store the semantics of metrics, so it is not possible to programmatically determine whether a metrics can be rate converted. The plugin contains a static list of metrics that should not be rate converted (gauges in Epic terminilogy).
Escalate Plugin
The Escalate plugin allows Trafﬁc Server to try an alternate origin when the origin server in the remap rule is either unavailable or returns speciﬁc HTTP error codes. Some services call this failover or fail-action.
Plugin Conﬁguration
The escalate plugin is a remap plugin (not global) and takes a parameter with two delimitated ﬁelds: comma-separated-error-codes:secondary-origin-server. For instance, @pparam=401,404,410,502:second-origin.example.com
3.7. Plugins 107 Apache-Trafﬁcserver-Server Documentation, latest would have Trafﬁc Server send a cache miss to second-origin.example.com when the origin server in the remap rule returns a 401, 404, 410, or 502 error code. @pparam=–pristine This option sends the “pristine” Host: header (eg, the Host: header that the client sent) to the escalated request.
Installation
This plugin is only built if the conﬁgure option
--enable-experimental-plugins is given at build time.
Example
With this line in remap.config map cdn.example.com origin.example.com @plugin=escalate.so @pparam=401,404,410,502:second-origin.example.com @pparam=--
˓→pristine
Trafﬁc Server would accept a request for cdn.example.com and, on a cache miss, proxy the request to origin. example.com. If the response code from that server is a 401, 404, 410, or 502, then Trafﬁc Server would proxy the request to second-origin.example.com, using a Host: header of cdn.example.com.
ESI Plugin
This plugin implements the ESI speciﬁcation.
Speciﬁcation
Supportted ESI tags: esi:include esi:remove esi:comment esi:vars esi:choose esi:when esi:otherwise esi:try esi:attempt esi:except  extended ESI tags: esi:special-include Supported variables:
$(HTTP_HOST) $(HTTP_REFERER) $(HTTP_ACCEPT_LANGUAGE{name})
108 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
$(HTTP_COOKIE{name}) or $(HTTP_COOKIE{name;subkey}) $(QUERY_STRING{name}) $(HTTP_HEADER{hdr_name})
Note: the name is the key name such as “username”, “id” etc. For cookie support sub-name or sub-key, the for- mat is: name;subkey, such as “l;u”, “l;t” etc. e.g. such cookie string: l=u=test&t=1350952328, the value of $(HTTP_COOKIE{“l;u”}) is test and the value of $(HTTP_COOKIE{“l;t”}) is 1350952328
Compile and Installation
This plugin is only built if the conﬁgure option
--enable-experimental-plugins is given at build time. Note that this plugin is built and installed in combination with the combo handler module, since they share common code.
Enabling ESI
1. First we need to set up /usr/local/etc/trafﬁcserver/plugin.conﬁg and make sure the following line is present. esi.so
2. There are four options you can add to the above. • “–private-response” will add private cache control and expires header to the processed ESI document. • “–packed-node-support” will enable the support for using packed node, which will improve the performance of parsing cached ESI document. • “–disable-gzip-output” will disable gzipped output, which will NOT gzip the output anyway. • “–ﬁrst-byte-ﬂush” will enable the ﬁrst byte ﬂush feature, which will ﬂush content to users as soon as the entire ESI document is received and parsed without all ESI includes fetched (the ﬂushing will stop at the ESI include markup till that include is fetched). 3. We need a mapping for origin server response that contains the ESI markup. Assume that the ATS server is abc.com. And your origin server is xyz.com and the response containing ESI markup is http://xyz.com/esi.php. We will need the following line in /usr/local/etc/trafﬁcserver/remap.conﬁg map http://abc.com/esi.php http://xyz.com/esi.php
4. Your response should contain ESI markup and a response header of ‘X-Esi: 1’. e.g. using PHP,
   Hello,   
5. You will need a mapping for the src of the ESI include in remap.conﬁg if it is not already present. map http://abc.com/date.php http://xyz.com/date.php
3.7. Plugins 109 Apache-Trafﬁcserver-Server Documentation, latest
Or if both your ESI response and the ESI include comes from the same origin server, you can have the following line in remap.conﬁg instead to replace separate map rules for date.php and esi.php map http://abc.com/ http://xyz.com/
6. Here is a sample PHP for date.php
Useful Note
1. You can provide proper cache control header and the ESI response and ESI include response can be cached separately. It is extremely useful for rendering page with multiple modules. The page layout can be a ESI response with multiple ESI include include, each for different module. The page layour ESI response can be cached and each individual ESI include can also be cached with different duration. 2. You might want to compile the code without using ESI_PACKED_NODE_SUPPORT because it may not work in some corner cases
Differences from Spec - http://www.w3.org/TR/esi-lang
1.  does not support “alt” and “onerror” attributes 2.  is not supported 3. You cannot have  inside another  4. HTTP_USER_AGENT variable is not supported 5. HTTP_COOKIE supports fetching for sub-key 6. HTTP_HEADER supports accessing request headers as variables
Generator Plugin
The Generator allows testing of synthetic workloads by generating HTTP responses of various sizes. The size and cacheability of the response is speciﬁed by the ﬁrst two coomponents of the requested URL path. This plugin only supports the GET and HEAD HTTP methods. Path component Description 1 cache or nocache. If cache is specifed, the Gener- 2 ator plugin will respond with Cache-Control head- ers marking the response as cacheable for 24 hours. In- tegral number of bytes to return in the response. Path components after the ﬁrst 2 are ignored. This means that the trailing path components can be manipulated to create unique URLs following any covenient convention. The Generator plugin publishes the following metrics: generator.response_bytes: The total number of bytes emitted generator.response_count: The number of HTTP responses generated by the plugin
110 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Examples:
The most common way to use the Generator plugin is to conﬁgure it as a remap plugin in remap.config: map http://workload.example.com/ http://127.0.0.1/\ @plugin=generator.so
Notice that although the remap target is never contacted because the Generator plugin intercepts the request and acts as the origin server, it must be syntactically valid and resolvable in DNS. A 10 byte, cacheable object can then be generated:
$ curl -o /dev/null -x 127.0.0.1:8080 http://workload.example.com/cache/10/
˓→caf1fc92332b3a3c8cb8b3826b6a1658
The Generator plugin can return responses as large as you like:
$ curl -o /dev/null -x 127.0.0.1:8080 http://workload.example.com/cache/$((10 * 1024 ˓→* 1024))/$RANDOM
GeoIP ACLs Plugin
This is a simple ATS plugin for denying (or allowing) requests based on the source IP geo-location. Currently only the Maxmind APIs are supported, but we’d be happy to other other (open) APIs if you let us know. This plugin comes with the standard distribution of Apache Trafﬁc Server, and should be installed as part of the normal build process.
Conﬁguration
Once installed, there are three primary use cases, which we will discuss in details. Note that in all conﬁgurations, the ﬁrst plugin parameter must specify what the matches should be applied to. Currently, only one rule set is supported, for Country ISO codes. This is speciﬁed with a parameter of
@pparam=country
Future additions to this plugin could include other regions, such as city, state, continent etc. The three typical use cases are as follows: 1. Per remap conﬁgurations, applicable to the entire remap rule. This is useful when you can partition your content so that entire preﬁx paths should be ﬁltered. For example, lets assume that http://example.com/music is restricted to US customers only, and everything else is world wide accessible. In remap.conﬁg, you would have something like
 map http://example.com/music http://music.example.com \ @plugin=geoip_acl.so @pparam=country @pparam=allow @pparam=US map http://example.com http://other.example.com
2. If you can not partition the data with a path preﬁx, you can specify a separate regex mapping ﬁlter. The remap.conﬁg ﬁle might then look like
 map http://example.com http://music.example.com \ @plugin=geoip_acl.so @pparam=country \ @pparam=regex::/etc/music.regex where music.regex is a format with PCRE (perl compatible) regular expressions, and unique rules for match. E.g.:
3.7. Plugins 111 Apache-Trafﬁcserver-Server Documentation, latest
.*\.mp3 allow US .*\.ogg deny US
Note that the default in the case of no matches on the regular expressions is to “allow” the request. This can be overriden, see next use case. 3. You can also combine 1) and 2), and provide defaults in the remap.conﬁg conﬁguration, which then applies for the cases where no regular expressions matches at all. This would be useful to override the default which is to allow all requests that don’t match. For example
 map http://example.com http://music.example.com \ @plugin=geoip_acl.so @pparam=country @pparam=allow @pparam=US @pparam=regex::/etc/music.regex
This tells the plugin that in the situation where there is no matching regular expression, only allow requests originating from the US. Finally, there’s one additional parameter option that can be used
@pparam=html::/some/path.html
This will override the default reponse body for the denied responses with a custom piece of HTML. This can be useful to explain to your users why they are getting denied access to a particular piece of content. This conﬁguration can be used with any of the use cases described above.
HIPES Plugin
This is a remap plugin used in the HIPES system.
Conﬁguration urlp: Default: url Name of the query parameter for the service URL path: Default: / Path to use for the service URL ssl Default: no Use SSL when connecting to the service service Service server, host[:port] server Default: hipes.yimg.com Name of HIPES server, host[:port] active_timeout The active connection timeout in ms no_activity_timeout The no activity timeout in ms connect_timeout The connect timeout in ms dns_timeout The DNS timeout The timeout options override the server defaults (from `records.config), and only apply to the connection to the speciﬁc “service” host.
Notes on HIPES
HTTP Pipes (aka HIPES and pronounced “Hippies”) allows data services to be pipelined together, as illustrated by the example below.
112 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Example
1. ATS is run on port 80 and apache HTTP web server is run on port 8080 on localhost (127.0.0.1) 2. The HIPES plugin is used in remap.config
 map http://127.0.0.1/svc_case http://nosuchhost @plugin=hipes.so
˓→@pparam=service:127.0.0.1:8080 @pparam=path:svc_case.php @pparam=server:127.0.0.
˓→1 map http://127.0.0.1/svc_reverse http://nosuchhost @plugin=hipes.so
˓→@pparam=service:127.0.0.1:8080 @pparam=path:svc_reverse.php @pparam=server:127.
˓→0.0.1 map http://127.0.0.1/test.txt http://127.0.0.1:8080/test.txt
3. The plugin remaps the incoming URL such as
 http://127.0.0.1/svc_reverse/svc_case;case=u/test.txt to the following http://127.0.0.1:8080/svc_reverse?url=http%3A%2F%2F127.0.0.1%2Fsvc_case%3Bcase%3Du
˓→%2Ftest.txt
4. The service svc_reverse.php fetches the url from the ATS again and the plugin remaps the URL
 http://127.0.0.1/svc_case;case=u/test.txt to this URL http://127.0.0.1:8080/svc_case.php?case=u&url=http%3A%2F%2F127.0.0.1%2Ftest.txt
5. In this example, the service svc_case.php fetches and transforms the response of http://127.0.0. 1/test.txt (which ATS proxies the request to a local ﬁle) to upper case. And subsequently the service svc_reverse.php receives the response and reverse the order before the response is sent back to the client by ATS.
Notes on reducing trafﬁc
There can be signiﬁcant overhead using HIPES because the data can traverse through ATS many times. Caching can be important to reduce trafﬁc to services/through ATS and can be achieved via a proper Cache-Control header returned by the services. Another way to reduce trafﬁc through ATS is to have ATS to return 302 redirects to url for the requests made by service, instead of proxying the requests to that url. However, the service must then be able to follow the redirect. The down side is that we cannot use ATS to cache intermediate results. Below is an example of using redirect.
Modiﬁcation to above example to reduce trafﬁc using redirect
1. The service svc_reverse.php is modiﬁed to add a header of X-HIPES-Redirect: 2 to the request made against url. 2. HIPES plugin will instruct ATS to return a redirect response to this url
 http://127.0.0.1:8080/svc_case.php?case=u&url=http%3A%2F%2F127.0.0.1%2Ftest.txt for the following request
3.7. Plugins 113 Apache-Trafﬁcserver-Server Documentation, latest
 http://127.0.0.1/svc_case;case=u/test.txt
3. The service svc_reverse.php is also modiﬁed to follow the redirect. Thus the response of the service of svc_case.php will not pass through ATS and will pass to svc_reverse.php service instead.
Memcache Plugin
This plugin implements the memcache protocol for cache contents.
Installation
Add the following line to plugin.config:
 tsmemcache.so 11211
In this case, the plugin will use the default behaviour: • Listen for memcache trafﬁc on the default memcache port 11211 • Write memcache data to the cache in the background (without waiting)
Conﬁguration
Alternatively, a conﬁguration can also be speciﬁed:
 tsmemcache.so/sample.tsmemcache.config
After modifying plugin.conﬁg, restart trafﬁc server (sudo trafﬁc_ctl server restart) the conﬁguration is also re-read when a management update is given (sudo trafﬁc_ctl conﬁg reload)
Options
Flags and options are: port: A single port to listen on for memcache protocol commands. Options in the code: TSMEMCACHE_WRITE_SYNC whether or not to wait for the write to complete.
Metalink Plugin
The Metalink plugin implements the Metalink download description format in order to try not to download the same ﬁle twice. This improves cache efﬁciency and speeds up users’ downloads. It takes standard headers and knowledge about objects in the cache and potentially rewrites those headers so that a client will use a URL that’s already cached instead of one that isn’t. The headers are speciﬁed in RFC 6249 (Met- alink/HTTP: Mirrors and Hashes) and RFC 3230 (Instance Digests in HTTP) and are sent by various download redi- rectors or content distribution networks. A lot of download sites distribute the same ﬁles from many different mirrors and users don’t know which mirrors are already cached. These sites often present users with a simple download button, but the button doesn’t predictably access the same mirror, or a mirror that’s already cached. To users it seems like the download works sometimes (takes seconds) and not others (takes hours), which is frustrating.
114 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
An extreme example of this happens when users share a limited, possibly unreliable internet connection, as is common in parts of Africa for example.
How it Works
When the plugin sees a response with a Location: ... header and a Digest: SHA-256=... header, it checks if the URL in the Location header is already cached. If it isn’t, then it tries to ﬁnd a URL that is cached to use instead. It looks in the cache for some object that matches the digest in the Digest header and if it succeeds, then it rewites the Location header with that object’s URL. This way a client should get sent to a URL that’s already cached and won’t download the ﬁle again.
Installation
The Metalink plugin is a global plugin. Enable it by adding metalink.so to your plugin.config ﬁle. There are no options.
Implementation Status
The plugin implements the TS_HTTP_SEND_RESPONSE_HDR_HOOK hook to check and potentially rewrite the Location and Digest headers after responses are cached. It doesn’t do it before they’re cached because the contents of the cache can change after responses are cached. It uses TSCacheRead() to check if the URL in the Location header is already cached. In future, the plugin should also check if the URL is fresh or not. The plugin implements the TS_HTTP_READ_RESPONSE_HDR_HOOK hook and a null transformation to com- pute the SHA-256 digest for content as it’s added to the cache. It uses SHA256_Init(), SHA256_Update(), and SHA256_Final() from OpenSSL to compute the digest, then it uses TSCacheWrite() to associate the digest with the request URL. This adds a new cache object where the key is the digest and the object is the request URL. To check if the cache already contains content that matches a digest, the plugin must call TSCacheRead() with the digest as the key, read the URL stored in the resultant object, and then call TSCacheRead() again with this URL as the key. This is probably inefﬁcient and should be improved. An early version of the plugin scanned Link: <...>; rel=duplicate headers. If the URL in the Location: ... header wasn’t already cached, it scanned Link: <...>; rel=duplicate headers for a URL that was. The Digest: SHA-256=... header is superior because it will ﬁnd content that already exists in the cache in every case that a Link: <...>; rel=duplicate header would, plus in cases where the URL is not listed among the Link: <...>; rel=duplicate headers, maybe because the content was downloaded from a URL not participating in the content distribution network, or maybe because there are too many mirrors to list in Link: <...>; rel=duplicate headers. The Digest: SHA-256=... header is also more efﬁcient than Link: <...>; rel=duplicate headers because it involves a constant number of cache lookups. RFC 6249 requires a Digest: SHA-256=... header or Link: <...>; rel=duplicate headers MUST be ignored: If Instance Digests are not provided by the Metalink servers, the Link header ﬁelds pertaining to this speciﬁcation MUST be ignored. Metalinks contain whole ﬁle hashes as described in Section 6, and MUST include SHA-256, as speciﬁed in [FIPS-180-3].
3.7. Plugins 115 Apache-Trafﬁcserver-Server Documentation, latest
MP4 Plugin
This module provides streaming media server support for MP4 ﬁles. User can send a HTTP request to the server with start argument which is measured in seconds, and the server will respond with the stream such that its start position corresponds to the requested time, for example: http://v.foo.com/dota2.mp4?start=290.12
This allows performing a random seeking at any time. We can use ﬂash player, vlc, mplayer, ﬁrefox or chrome to play the streaming media. This plugin can be used as a remap plugin. We can write this in remap.conﬁg: map http://v.foo.com/ http://v.internal.com/ @plugin=mp4.so
Note
This plugin requires that the moov box in the mp4 ﬁle should be ahead of mdat box. It is not a good idea to cache a large mp4 ﬁle, many video sites will cut a large video ﬁle into many small mp4 ﬁles, and each small mp4 ﬁle will be less than 80M(bytes), it will be a reasonable choice.
MySQL Remap Plugin
This is a basic plugin for doing dynamic “remaps” from a database. It essentially rewrites the incoming request’s Host header / origin server connection to one retrieved from a database. The generic proxying setup is the following:
UA----> Traffic Server----> Origin Server
Without the plugin a request like:
GET/path/to/something HTTP/1.1 Host: original.host.com
Ends up requesting http://original.host.com/path/to/something With this plugin enabled, you can easily change that to anywhere you desire. Imagine the many possibilities.... We have benchmarked the plugin with ab at about 9200 requests/sec (1.7k object) on a commodity hardware with a local setup of both, MySQL and Trafﬁc Server local. Real performance is likely to be substantially higher, up to the MySQL’s max queries / second.
Installation
This plugin is only built if the conﬁgure option
--enable-experimental-plugins is given at build time.
116 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Conﬁguration
Import the default schema to a database you create: mysql-u root-p-e"CREATE DATABASE mysql_remap;" # create a new database mysql-u root-p mysql_remap< schema/import.sql # import the provided schema insert some interesting values in mysql_remap.hostname & mysql_remap.map Trafﬁc Server plugin conﬁguration is done inside a global conﬁguration ﬁle: /etc/trafficserver/plugin. config: mysql_remap.so/etc/trafficserver/mysql_remap.ini
The INI ﬁle should contain the following values:
[mysql_remap] mysql_host= localhost #default mysql_port= 3306 #default mysql_username= remap_user mysql_password= mysql_database= mysql_remap #default
To debug errors, start trafﬁcserver manually using: traffic_server-T"mysql_remap"
And resolve any errors or warnings displayed.
Signed URL Plugin
This plugin checks a signature query string on a URL and rejects (HTTP 403) or redirects (HTTP 302) when the check fails. The signature is based on a secret key that both a signing portal and the Trafﬁc Server cache share. The algorithm for the signature may be either MD5 or SHA1. When the signature check passes, the query string of the request is stripped and continues to process as if there were no query string at all.
Installation
To make this plugin available, you must either enable experimental plugins when building Trafﬁc Server:
./configure--enable-experimental-plugins
Or use tsxs to compile the plugin against your current Trafﬁc Server build. To do this, you must ensure that: 1. Development packages for Trafﬁc Server are installed. 2. The tsxs binary is in your path. 3. The version of this plugin you are building, and the version of Trafﬁc Server against which you are building it are compatible. Once those conditions are satisﬁed, enter the source directory for the plugin and perform the following: make-f Makefile.tsxs make-f Makefile.tsxs install
3.7. Plugins 117 Apache-Trafﬁcserver-Server Documentation, latest
Conﬁguration
Conﬁguring URL signature veriﬁcation within Trafﬁc Server using this plugin is a two step process. First, you must generate a conﬁguration ﬁle containing the list of valid signing keys. Secondly, you must indicate to Trafﬁc Server which URLs require valid signatures.
Generating Keys
This plugin comes with two Perl scripts which assist in generating signatures. For Trafﬁc Server to verify URL signatures, it must have the relevant keys. Using the provided genkeys script, you can generate a suitable conﬁguration ﬁle:
./genkeys.pl> url_sig.config
The resulting ﬁle will look something like the following, with the actual keys differing (as they are generated randomly each time the script is run): key0= YwG7iAxDo6Gaa38KJOceV4nsxiAJZ3DS key1= nLE3SZKRgaNM9hLz_HnIvrCw_GtTUJT1 key2= YicZbmr6KlxfxPTJ3p9vYhARdPQ9WJYZ key3= DTV4Tcn046eM9BzJMeYrYpm3kbqOtBs7 key4= C1r6R6MINoQd5YSH25fU66tuRhhz3fs_ key5= l4dxe6YEpYbJtyiOmX5mafhwKImC5kej key6= ekKNHXu9_oOC5eqIGJVxV0vI9FYvKVya key7= BrjibTmpTTuhMHqphkQAuCWA0Zg97WQB key8= rEtWLb1jcYoq9VG8Z8TKgX4GxBuro20J key9= mrP_6ibDBG4iYpfDB6W8yn3ZyGmdwc6M key10= tbzoTTGZXPLcvpswCQCYz1DAIZcAOGyX key11= lWsn6gUeSEW79Fk2kwKVfzhVG87EXLna key12= Riox0SmGtBWsrieLUHVWtpj18STM4MP1 key13= kBsn332B7yG3HdcV7Tw51pkvHod7_84l key14= hYI4GUoIlZRf0AyuXkT3QLvBMEoFxkma key15= EIgJKwIR0LU9CUeTUdVtjMgGmxeCIbdg error_url= 403
This ﬁle should be placed in your Trafﬁc Server etc directory, with permissions and ownership set such that only the Trafﬁc Server processes may read it.
: The conﬁguration ﬁle contains the full set of secret keys which Trafﬁc Server will be using to verify incoming requests, and as such should be treated with as much care as any other ﬁle in your infrastructure containing keys, pass phrases, and other sensitive data. Unauthorized access to the contents of this ﬁle will allow others to spoof requests from your signing portal, thus defeating the entire purpose of using a signing portal in the ﬁrst place.
Requiring Signatures on URLs
To require a valid signature, veriﬁed by a key from the list you generated earlier, modify your remap.config conﬁguration to include this plugin for any rules you wish it to affect. Two parameters for each remap rule are required:
@plugin=url_sig.so @pparam=
118 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
The ﬁrst simply enables this plugin for the rule. The second speciﬁes the location of the conﬁguration ﬁle containing your signing keys. For example, if we wanted to restrict all paths under a /download directory on our website foo.com we might have a remap line like this:
 map http://foo.com/download/ http://origin.server.tld/download/\ @plugin=url_sig.so @pparam=url_sig.config
Signing a URL
Signing a URL is solely the responsibility of your signing portal service. This requires that whatever application runs that service must also have a list of your signing keys (generated earlier in Conﬁguration and stored in the url_sig. config ﬁle in your Trafﬁc Server conﬁguration directory). How your signing portal application is informed about, or stores, these keys is up to you, but it is critical that the keyN index numbers are matched to the same keys. Signing is performed by adding several query parameters to a URL, before redirecting the client. The parameters’ values are all generated by your signing portal application, and then a hash is calculated by your portal, using the entire URL just constructed, and attached as the ﬁnal query parameter.
: Ordering is important when adding the query parameters and generating the signature. The signature itself is a hash, using your chosen algorithm, of the entire URL to which you are about to redirect the client, up to and including the S= substring indicating the signature parameter.
The following list details all the query parameters you must add to the URL you will hand back to the client for redirection. Client IP The IP address of the client being redirected. This must be their IP as it will appear to your Trafﬁc Server cache:
C=
Expiration The time at which this signature will no longer be valid, expressed in seconds since epoch (and thus in UTC):
E=
Algorithm The hash algorithm which your signing portal application has elected to use for this signature:
A=
The only supported values at this time are: Value Algorithm 1 HMAC_SHA1 2 HMAC_MD5 Key Index The key number, from your plugin conﬁguration, which was used for this signature. See Conﬁguration for generating these keys and determining the index number of each:
K=
Parts Conﬁgures which components of the URL to use for signature veriﬁcation. The value of this paramerts is a string of ones and zeroes, each enabling or disabling the use of a URL part for signatures. The URL scheme (e.g. http://) is never part of the signature. The ﬁrst number of this parameter’s value indicates whether to
3.7. Plugins 119 Apache-Trafﬁcserver-Server Documentation, latest
 include the FQDN, and all remaining numbers determine whether to use the directory parts of the URL. If there are more directories in the URL than there are numbers in this parameter, the last number is repeated as many times as necessary:
P=
Examples include: Value Effect 1 Use the FQDN and all directory parts for signature veriﬁcation. 01 Ignore the FQDN, but verify using all directory parts. 0110 Ignore the FQDN, and use only the ﬁrst two directory parts, skipping the remainder, for signatures. 110 Use the FQDN and ﬁrst directory for signature veriﬁcation, but ignore the remainder of the path. Signature The actual signature hash:
S=
The hash should be calculated in accordance with the parts speciﬁcation you have declared in the P= query parameter, which if you have chosen any value other than 1 may require additional URL parsing be performed by your signing portal. Additionally, all query parameters up to and including the S= substring for this parameter must be included, and must be in the same order as they are returned to the client for redirection. For obvious reasons, the value of this parameter is not included in the source string being hashed. As a simple example, if we are about to redirect a client to the URL https://foo.com/downloads/ expensive-app.exe with signature veriﬁcation enabled, then we will compute a signature on the following string:
 foo.com/downloads/expensive-app.exe?C=1.2.3.4&E=1453846938&A=1&K=2&P=1&S=
And, assuming that key2 from our secrets ﬁle matches our example in Conﬁguration, then our signature be- comes:
8c5cfa440458233452ee9b5b570063a0e71827f2
Which is then appended to the URL for redirection as the value of the S parameter. For an example implementation of signing which may be adapted for your own portal, refer to the ﬁle sign.pl included with the source code of this plugin.
Edge Cache Debugging
To include debugging output for this plugin in your Trafﬁc Server logs, adjust the values for :ts:cv:‘proxy.conﬁg.diags.debug.enabled‘ and :ts:cv:‘proxy.conﬁg.diags.debug.tags‘ in your records.config as so:
CONFIG proxy.config.diags.debug.enabled INT1 CONFIG proxy.config.diags.debug.tags STRING url_sig
Once updated, issue a traffic_ctl config reload to make the settings active.
Example
1. Enable experimental plugins when building Trafﬁc Server:
120 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
./configure--enable-experimental-plugins
2. Generate a secrets conﬁguration for Trafﬁc Server (replacing the output location with something appropriate to your Trafﬁc Server installation):
 genkeys.pl>/usr/local/trafficserver/etc/trafficserver/url_sig.config
3. Verify that your conﬁguration looks like the following, with actual key values altered:
 key0= YwG7iAxDo6Gaa38KJOceV4nsxiAJZ3DS key1= nLE3SZKRgaNM9hLz_HnIvrCw_GtTUJT1 key2= YicZbmr6KlxfxPTJ3p9vYhARdPQ9WJYZ key3= DTV4Tcn046eM9BzJMeYrYpm3kbqOtBs7 key4= C1r6R6MINoQd5YSH25fU66tuRhhz3fs_ key5= l4dxe6YEpYbJtyiOmX5mafhwKImC5kej key6= ekKNHXu9_oOC5eqIGJVxV0vI9FYvKVya key7= BrjibTmpTTuhMHqphkQAuCWA0Zg97WQB key8= rEtWLb1jcYoq9VG8Z8TKgX4GxBuro20J key9= mrP_6ibDBG4iYpfDB6W8yn3ZyGmdwc6M key10= tbzoTTGZXPLcvpswCQCYz1DAIZcAOGyX key11= lWsn6gUeSEW79Fk2kwKVfzhVG87EXLna key12= Riox0SmGtBWsrieLUHVWtpj18STM4MP1 key13= kBsn332B7yG3HdcV7Tw51pkvHod7_84l key14= hYI4GUoIlZRf0AyuXkT3QLvBMEoFxkma key15= EIgJKwIR0LU9CUeTUdVtjMgGmxeCIbdg error_url= 403
4. Enable signature veriﬁcation for a remap rule in Trafﬁc Server by modifying remap.config (here we will just remap to Google’s homepage for demonstrative purposes):
 map http://test-remap.domain.com/download/foo http://google.com \ @plugin=url_sig.so @pparam=url_sig.config
5. Reload your Trafﬁc Server conﬁguration to ensure the changes are active:
 traffic_ctl config reload
6. Attempt to access the now-protected URL without a valid signature. This will fail, and that is a good thing, as it demonstrates that Trafﬁc Server now rejects any requests to paths matching that rule which do not include a valid signature.:
$ curl -vs -H'Host: test-remap.domain.com' http://localhost:8080/download/foo * Adding handle: conn: 0x200f8a0 * Adding handle: send: 0 * Adding handle: recv: 0 * Curl_addHandleToPipeline: length: 1 * - Conn 0 (0x200f8a0) send_pipe: 1, recv_pipe: 0 * About to connect() to localhost port 8080 (#0) * Trying 127.0.0.1... * Connected to localhost (127.0.0.1) port 8080 (#0) > GET /download/foo HTTP/1.1 > User-Agent: curl/7.32.0 > Accept: */* > Host: test-remap.domain.com > < HTTP/1.1 403 Forbidden < Date: Tue, 15 Apr 2014 22:57:32 GMT
3.7. Plugins 121 Apache-Trafﬁcserver-Server Documentation, latest
< Connection: close * Server ATS/5.0.0 is not blacklisted < Server: ATS/5.0.0 < Cache-Control: no-store < Content-Type: text/plain < Content-Language: en < Content-Length: 21 < * Closing connection 0 Authorization Denied$
7. Generate a signed URL using the included sign.pl script:
 sign.pl--url http://test-remap.domain.com/download/foo \ --useparts1--algorithm1--duration 60--keyindex3\ --key DTV4Tcn046eM9BzJMeYrYpm3kbqOtBs7
8. Now access the protected URL with a valid signature:
$ curl -s -o /dev/null -v --max-redirs 0 -H 'Host: test-remap.domain.com' \ 'http://test-remap.domain.com/download/foo?E=1453848506&A=1&K=3&P=1&
˓→S=7aea86592de3e9c1b05771b2538a30956c6f10a3' * Adding handle: conn: 0xef0a90 * Adding handle: send: 0 * Adding handle: recv: 0 * Curl_addHandleToPipeline: length: 1 * - Conn 0 (0xef0a90) send_pipe: 1, recv_pipe: 0 * About to connect() to localhost port 8080 (#0) * Trying 127.0.0.1... * Connected to localhost (127.0.0.1) port 8080 (#0) > GET /download/foo?E=1397603088&A=1&K=3&P=1&
˓→S=28d822f68ac7265db61a8441e0877a98fe1007cc HTTP/1.1 > User-Agent: curl/7.32.0 > Accept: */* > Host: test-remap.domain.com > < HTTP/1.1 200 OK < Location: http://www.google.com/ < Content-Type: text/html; charset=UTF-8 < Date: Tue, 15 Apr 2014 23:04:36 GMT < Expires: Thu, 15 May 2014 23:04:36 GMT < Cache-Control: public, max-age=2592000 * Server ATS/5.0.0 is not blacklisted < Server: ATS/5.0.0 < Content-Length: 219 < X-XSS-Protection: 1; mode=block < X-Frame-Options: SAMEORIGIN < Alternate-Protocol: 80:quic < Age: 0 < Connection: keep-alive < { [data not shown] * Connection #0 to host localhost left intact
122 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
SSL Headers Plugin
The sslheaders plugins injects SSL session information into HTTP request headers. It can operate as a global plugin or as a remap plugin.
Plugin Options
The following options may be speciﬁed when loading the plugin in plugin.config or remap.config: --attach=WHICH This option speciﬁes which HTTP request the SSL headers are attached to. client causes the headers to be injected into the client request. This is primarily useful if another plugin should inspect then. server is the default and injects the headers into the origin server request. both injects the headers into both the client request and the origin server request. A list of KEY=VALUE pairs follows any options. The KEY names the HTTP header to inject, and VALUE names the SSL session ﬁeld. SSL session ﬁeld Description client.certiﬁcate The client certiﬁcate in PEM format client.subject The client certiﬁcate subject DN client.issuer The client certiﬁcate issuer DN client.serial The client certiﬁcate serial number in hexadecimal format client.signature The client certiﬁcate signature in hexadecimal format client.notbefore The client certiﬁcate validity start time client.notafter The client certiﬁcate validity end time server.certiﬁcate The server certiﬁcate in PEM format server.subject The server certiﬁcate subject DN server.issuer The server certiﬁcate issuer DN server.serial The server certiﬁcate serial number in hexadecimal format server.signature The server certiﬁcate signature in hexadecimal format server.notbefore The server certiﬁcate validity start time server.notafter The server certiﬁcate validity end time The client.certiﬁcate and server.certiﬁcate ﬁelds emit the corresponding certiﬁcate in PEM format, with newline char- acters replaced by spaces. If the sslheaders plugin activtes on non-SSL connections, it will delete all the conﬁgured HTTP header names so that malicious clients cannot inject misleading information. If any of the SSL ﬁelds expand to an empty string, those headers are also deleted.
Examples:
In this example, the origin server is interested in the subject of the server certiﬁcate that was used to accept a client connetion. We can apply the sslheaders plugin to a generic remap rule to provide this information. The remap. config conﬁguration would be: regex_map https://*.example.com/ http://origin.example.com/\ @plugin=sslheaders.so @pparam=SSL-Server=server.subject
In this example, we have set :ts:cv:‘proxy.conﬁg.ssl.client.certiﬁcation_level‘ to request SSL client certiﬁcates. We can then conﬁgure sslheaders to populate the client certiﬁcate subject globally by adding it to plugin.config:
3.7. Plugins 123 Apache-Trafﬁcserver-Server Documentation, latest
 sslheaders.so SSL-Client-ID=client.subject SSL-Client-NotBefore=client.notbefore SSL-
˓→Client-NotAfter-client.notafter
Stale While Revalidate Plugin refresh content asynchronously while serving stale data
TS Lua Plugin
This module embeds Lua, via the standard Lua 5.1 interpreter, into Apache Trafﬁc Server™. With this module, we can implement ATS plugin by writing Lua script instead of C code. Lua code executed using this module can be 100% non-blocking because the powerful Lua coroutines have been integrated into the ATS event model.
Synopsis test_hdr.lua function send_response() ts.client_response.header['Rhost']= ts.ctx['rhost'] return 0 end function do_remap() local req_host= ts.client_request.header.Host ts.ctx['rhost']= string.reverse(req_host) ts.hook(TS_LUA_HOOK_SEND_RESPONSE_HDR, send_response) return 0 end sethost.lua local HOSTNAME='' function __init__(argtb) if (#argtb) < 1 then print(argtb[0],'hostname parameter required!!') return -1 end HOSTNAME= argtb[1] end function do_remap() ts.client_request.header['Host']= HOSTNAME return 0 end
Installation
This plugin is only built if the conﬁgure option
--enable-experimental-plugins is given at build time.
124 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Conﬁguration
This module acts as remap plugin of Trafﬁc Server, so we should realize ‘do_remap’ function in each lua script. We can write this in remap.conﬁg: map http://a.tbcdn.cn/ http://inner.tbcdn.cn/ @plugin=/XXX/tslua.so @pparam=/XXX/test_
˓→hdr.lua
Sometimes we want to receive parameters and process them in the script, we should realize ‘__init__’ function in the lua script, and we can write this in remap.conﬁg: map http://a.x.cn/ http://b.x.cn/ @plugin=/X/tslua.so @pparam=/X/sethost.lua
˓→@pparam=a.st.cn
This module can also act as a global plugin of Trafﬁc Server. In this case we should provide one of these functions in each lua script: • ‘do_global_txn_start’ • ‘do_global_txn_close’ • ‘do_global_os_dns’ • ‘do_global_pre_remap’ • ‘do_global_post_remap’ • ‘do_global_read_request’ • ‘do_global_send_request’ • ‘do_global_read_response’ • ‘do_global_send_response’ • ‘do_global_cache_lookup_complete’ • ‘do_global_read_cache’ • ‘do_global_select_alt’ We can write this in plugin.conﬁg: tslua.so/etc/trafficserver/script/test_global_hdr.lua
TS API for Lua
Introduction
The API is exposed to Lua in the form of one standard packages ts. This package is in the default global scope and is always available within lua script. This package can be introduced into Lua like this: ts.say('Hello World') ts.sleep(10)
TOP
3.7. Plugins 125 Apache-Trafﬁcserver-Server Documentation, latest ts.now syntax: val = ts.now() context: global description: This function returns the time since the Epoch (00:00:00 UTC, January 1, 1970), measured in seconds. It includes milliseconds as the decimal part. Here is an example: local nt= ts.now()-- 1395221053.123
TOP ts.debug syntax: ts.debug(MESSAGE) context: global description: Log the MESSAGE to trafﬁc.out if debug is enabled. Here is an example: ts.debug('I am in do_remap now.')
The debug tag is ts_lua and we should write this in records.conﬁg: CONFIG proxy.config.diags.debug.tags STRING ts_lua TOP
Remap status constants context: do_remap
TS_LUA_REMAP_NO_REMAP (0) TS_LUA_REMAP_DID_REMAP (1) TS_LUA_REMAP_NO_REMAP_STOP (2) TS_LUA_REMAP_DID_REMAP_STOP (3) TS_LUA_REMAP_ERROR (-1)
These constants are usually used as return value of do_remap function. TOP ts.remap.get_to_url_host syntax: ts.remap.get_to_url_host() context: do_remap description: retrieve the “to” host of the remap rule Here is an example:
126 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 function do_remap() local to_host= ts.remap.get_to_url_host() ts.debug(to_host) return 0 end
TOP ts.remap.get_to_url_port syntax: ts.remap.get_to_url_port() context: do_remap description: retrieve the “to” port of the remap rule TOP ts.remap.get_to_url_scheme syntax: ts.remap.get_to_url_scheme() context: do_remap description: retrieve the “to” scheme of the remap rule TOP ts.remap.get_to_uri syntax: ts.remap.get_to_uri() context: do_remap description: retrieve the “to” path of the remap rule TOP ts.remap.get_to_url syntax: ts.remap.get_to_url() context: do_remap description: retrieve the “to” url of the remap rule TOP ts.remap.get_from_url_host syntax: ts.remap.get_from_url_host() context: do_remap description: retrieve the “from” host of the remap rule TOP
3.7. Plugins 127 Apache-Trafﬁcserver-Server Documentation, latest ts.remap.get_from_url_port syntax: ts.remap.get_from_url_port() context: do_remap description: retrieve the “from” port of the remap rule TOP ts.remap.get_from_url_scheme syntax: ts.remap.get_from_url_scheme() context: do_remap description: retrieve the “from” scheme of the remap rule TOP ts.remap.get_from_uri syntax: ts.remap.get_from_uri() context: do_remap description: retrieve the “from” path of the remap rule TOP ts.remap.get_from_url syntax: ts.remap.get_from_url() context: do_remap description: retrieve the “from” url of the remap rule TOP ts.hook syntax: ts.hook(HOOK_POINT, FUNCTION) context: global or do_remap or do_global_* or later description: Hooks are points in http transaction processing where we can step in and do some work. FUNCTION will be called when the http transaction steps in to HOOK_POINT. Here is an example function send_response() s.client_response.header['SHE']='belief' end function do_remap() ts.hook(TS_LUA_HOOK_SEND_RESPONSE_HDR, send_response) end
128 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Then the client will get the response like this:
HTTP/1.1 200OK Content-Type: text/html Server: ATS/3.2.0 SHE: belief Connection: Keep-Alive ...
You can create global hook as well function do_some_work() ts.debug('do_some_work') return 0 end ts.hook(TS_LUA_HOOK_READ_REQUEST_HDR, do_some_work)
Or you can do it this way ts.hook(TS_LUA_HOOK_READ_REQUEST_HDR, function() ts.debug('do_some_work') return 0 end )
Also the return value of the function will control how the transaction will be re-enabled. Return value of 0 will cause the transaction to be re-enabled normally (TS_EVENT_HTTP_CONTINUE). Return value of 1 will be using TS_EVENT_HTTP_ERROR instead. TOP
Hook point constants
TS_LUA_HOOK_OS_DNS TS_LUA_HOOK_PRE_REMAP TS_LUA_HOOK_READ_CACHE_HDR TS_LUA_HOOK_SELECT_ALT TS_LUA_HOOK_TXN_CLOSE TS_LUA_HOOK_POST_REMAP TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE TS_LUA_HOOK_READ_REQUEST_HDR TS_LUA_HOOK_SEND_REQUEST_HDR TS_LUA_HOOK_READ_RESPONSE_HDR TS_LUA_HOOK_SEND_RESPONSE_HDR TS_LUA_REQUEST_TRANSFORM TS_LUA_RESPONSE_TRANSFORM
These constants are usually used in ts.hook method call. Additional Information:
3.7. Plugins 129 Apache-Trafﬁcserver-Server Documentation, latest
Hook Point Lua Hook Point Hook function be Hook function be constant registered within registered within global do_remap() via ts.hook()? context via ts.hook()? TS_HTTP_TXN_START_HOOKTS_LUA_HOOK_TXN_STARTNO YES TS_HTTP_READ_REQUEST_HDR_HOOKTS_LUA_HOOK_READ_REQUEST_HDRNO YES TS_HTTP_PRE_REMAP_HOOKTS_LUA_HOOK_PRE_REMAPNO YES TS_HTTP_POST_REMAP_HOOKTS_LUA_HOOK_POST_REMAPYES YES TS_HTTP_SELECT_ALT_HOOKTS_LUA_HOOK_SELECT_ALTNO NO TS_HTTP_READ_CACHE_HDR_HOOKTS_LUA_HOOK_READ_CACHE_HDRYES YES TS_HTTP_OS_DNS_HOOKTS_LUA_HOOK_OS_DNSYES YES TS_HTTP TS_LUA_HOOK YES YES _CACHE_LOOKUP_COMPLETE_HOOK_CACHE_LOOKUP_COMPLETE TS_HTTP TS_LUA_HOOK YES YES _SEND_REQUEST_HDR_HOOK_SEND_REQUEST_HDR TS_HTTP TS_LUA_HOOK YES YES _READ_RESPONSE_HDR_HOOK_READ_RESPONSE_HDR TS_HTTP TS_LUA_HOOK YES YES _SEND_RESPONSE_HDR_HOOK_SEND_RESPONSE_HDR TS_HTTP _RE- TS_LUA_REQUEST_TRANSFORMYES YES QUEST_TRANSFORM_HOOK TS_HTTP _RE- TS_LUA_RESPONSE_TRANSFORMYES YES SPONSE_TRANSFORM_HOOK TS_HTTP_TXN_CLOSE_HOOKTS_LUA_HOOK_TXN_CLOSEYES YES TOP ts.ctx syntax: ts.ctx[KEY] = VALUE syntax: VALUE = ts.ctx[KEY] context: do_remap or do_global_* or later description: This table can be used to store per-request Lua context data and has a life time identical to the current request. Here is an example: function send_response() ts.client_response.header['F-Header']= ts.ctx['hdr'] return 0 end function do_remap() ts.ctx['hdr']='foo' ts.hook(TS_LUA_HOOK_SEND_RESPONSE_HDR, send_response) return 0 end
Then the client will get the response like this:
HTTP/1.1 200OK Content-Type: text/html Server: ATS/3.2.0 F-Header: foo
130 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Connection: Keep-Alive ...
TOP ts.client_request.get_method syntax: ts.client_request.get_method() context: do_remap or do_global_* or later description: This function can be used to retrieve the current client request’s method name. String like “GET” or “POST” is returned. TOP ts.client_request.set_method syntax: ts.client_request.set_method() context: do_remap or do_global_* description: This function can be used to override the current client request’s method with METHOD_NAME. ts.client_request.get_version syntax: ver = ts.client_request.get_version() context: do_remap or do_global_* or later description: Return the http version string of the client request. Current possible values are 1.0, 1.1, and 0.9. TOP ts.client_request.set_version syntax: ts.client_request.set_version(VERSION_STR) context: do_remap or do_global_* or later description: Set the http version of the client request with the VERSION_STR ts.client_request.set_version('1.0')
TOP ts.client_request.get_uri syntax: ts.client_request.get_uri() context: do_remap or later description: This function can be used to retrieve the client request’s path.
3.7. Plugins 131 Apache-Trafﬁcserver-Server Documentation, latest
Here is an example: function do_remap() local uri= ts.client_request.get_uri() print(uri) end
Then GET /st?a=1 will yield the output: /st TOP ts.client_request.set_uri syntax: ts.client_request.set_uri(PATH) context: do_remap or do_global_* description: This function can be used to override the client request’s path. The PATH argument must be a Lua string and starts with / TOP ts.client_request.get_uri_args syntax: ts.client_request.get_uri_args() context: do_remap or do_global_* or later description: This function can be used to retrieve the client request’s query string. Here is an example: function do_remap() local query= ts.client_request.get_uri_args() print(query) end
Then GET /st?a=1&b=2 will yield the output: a=1&b=2 TOP ts.client_request.set_uri_args syntax: ts.client_request.set_uri_args(QUERY_STRING) context: do_remap or do_global_* description: This function can be used to override the client request’s query string. ts.client_request.set_uri_args('n=6&p=7')
TOP
132 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest ts.client_request.get_uri_params syntax: ts.client_request.get_uri_params() context: do_remap or do_global_* or later description: This function can be used to retrieve the client request’s parameter string. Here is an example: function do_remap() local query= ts.client_request.get_uri_params() print(query) end
Then GET /st;a=1 will yield the output: a=1 TOP ts.client_request.set_uri_params syntax: ts.client_request.set_uri_params(PARAMETER_STRING) context: do_remap or do_global_* description: This function can be used to override the client request’s parameter string. ts.client_request.set_uri_params('n=6')
TOP ts.client_request.get_url syntax: ts.client_request.get_url() context: do_remap or do_global_* or later description: This function can be used to retrieve the whole client request’s url. Here is an example: function do_remap() local url= ts.client_request.get_url() print(url) end
Then GET /st?a=1&b=2 HTTP/1.1\r\nHost: a.tbcdn.cn\r\n... will yield the output: http://a.tbcdn.cn/st?a=1&b=2 TOP ts.client_request.header.HEADER syntax: ts.client_request.header.HEADER = VALUE
3.7. Plugins 133 Apache-Trafﬁcserver-Server Documentation, latest
 syntax: ts.client_request.header[HEADER] = VALUE syntax: VALUE = ts.client_request.header.HEADER context: do_remap or do_global_* or later description: Set, add to, clear or get the current client request’s HEADER. Here is an example:
 function do_remap() local ua= ts.client_request.header['User-Agent'] print(ua) ts.client_request.header['Host']='a.tbcdn.cn' end
Then GET /st HTTP/1.1\r\nHost: b.tb.cn\r\nUser-Agent: Mozilla/5.0\r\n... will yield the output: Mozilla/5.0 TOP ts.client_request.get_headers syntax: ts.client_request.get_headers() context: do_remap or do_global_* or later description: Returns a Lua table holding all the headers for the current client request. Here is an example:
 function do_remap() hdrs= ts.client_request.get_headers() for k, v in pairs(hdrs) do print(k..':'..v) end end
Then GET /st HTTP/1.1\r\nHost: b.tb.cn\r\nUser-Aget: Mozilla/5.0\r\nAccept: */* will yield the output
Host: b.tb.cn User-Agent: Mozilla/5.0 Accept: */*
TOP
 ts.client_request.client_addr.get_addr
 syntax: ts.client_request.client_addr.get_addr() context: do_remap or do_global_* or later description: This function can be used to get socket address of the client. The ts.client_request.client_addr.get_addr function returns three values, ip is a string, port and family is number. Here is an example:
134 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 function do_remap() ip, port, family= ts.client_request.client_addr.get_addr() print(ip)-- 192.168.231.17 print(port)-- 17786 print(family)--2(AF_INET) return 0 end
TOP ts.client_request.client_addr.get_incoming_port syntax: ts.client_request.client_addr.get_incoming_port() context: do_remap or do_global_* or later description: This function can be used to get incoming port of the request. The ts.client_request.client_addr.get_incoming_port function returns incoming port as number. Here is an example: function do_global_read_request() port= ts.client_request.client_addr.get_incoming_port() print(port)-- 80 end
TOP ts.client_request.get_url_host syntax: host = ts.client_request.get_url_host() context: do_remap or do_global_* or later description: Return the host ﬁeld of the request url. Here is an example: function do_remap() local url_host= ts.client_request.get_url_host() print(url_host) end
Then GET /liuyurou.txt HTTP/1.1\r\nHost: 192.168.231.129:8080\r\n... will yield the output: 192.168.231.129 TOP ts.client_request.set_url_host syntax: ts.client_request.set_url_host(str) context: do_remap or do_global_*
3.7. Plugins 135 Apache-Trafﬁcserver-Server Documentation, latest
 description: Set host ﬁeld of the request url with str. This function is used to change the address of the origin server, and we should return TS_LUA_REMAP_DID_REMAP(_STOP) in do_remap. Here is an example:
 function do_remap() ts.client_request.set_url_host('192.168.231.130') ts.client_request.set_url_port(80) ts.client_request.set_url_scheme('http') return TS_LUA_REMAP_DID_REMAP end
 remap.conﬁg like this:
 map http://192.168.231.129:8080/ http://192.168.231.129:9999/
Then server request will connect to 192.168.231.130:80 TOP
 ts.client_request.get_url_port
 syntax: port = ts.client_request.get_url_port() context: do_remap or do_global_* or later description: Returns the port ﬁeld of the request url as a Lua number. Here is an example:
 function do_remap() local url_port= ts.client_request.get_url_port() print(url_port) end
Then Then GET /liuyurou.txt HTTP/1.1\r\nHost: 192.168.231.129:8080\r\n... will yield the output: 8080 TOP
 ts.client_request.set_url_port
 syntax: ts.client_request.set_url_port(NUMBER) context: do_remap or do_global_* description: Set port ﬁeld of the request url with NUMBER. This function is used to change the address of the origin server, and we should return TS_LUA_REMAP_DID_REMAP(_STOP) in do_remap. TOP ts.client_request.get_url_scheme syntax: scheme = ts.client_request.get_url_scheme() context: do_remap or do_global_* or later
136 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest description: Return the scheme ﬁeld of the request url. Here is an example: function do_remap() local url_scheme= ts.client_request.get_url_scheme() print(url_scheme) end
Then GET /liuyurou.txt HTTP/1.1\r\nHost: 192.168.231.129:8080\r\n... will yield the output: http TOP ts.client_request.set_url_scheme syntax: ts.client_request.set_url_scheme(str) context: do_remap or do_global_* description: Set scheme ﬁeld of the request url with str. This function is used to change the scheme of the server request, and we should return TS_LUA_REMAP_DID_REMAP(_STOP) in do_remap. TOP ts.http.set_cache_url syntax: ts.http.set_cache_url(KEY_URL) context: do_remap or do_global_* description: This function can be used to modify the cache key for the client request. Here is an example: function do_remap() ts.http.set_cache_url('http://a.tbcdn.cn/foo') return 0 end
TOP ts.http.get_cache_lookup_url syntax: ts.http.get_cache_lookup_url() context: do_global_cache_lookup_complete description: This function can be used to get the cache lookup url for the client request. Here is an example function cache_lookup() ts.http.set_cache_lookup_url('http://bad.com/bad.html') local cache= ts.http.get_cache_lookup_url() ts.debug(cache)
3.7. Plugins 137 Apache-Trafﬁcserver-Server Documentation, latest
 end function do_remap() ts.hook(TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE, cache_lookup) return 0 end
TOP ts.http.set_cache_lookup_url syntax: ts.http.set_cache_lookup_url() context: do_global_cache_lookup_complete description: This function can be used to set the cache lookup url for the client request. TOP ts.http.set_server_resp_no_store syntax: ts.http.set_server_resp_no_store(status) context: do_global_read_response description: This function can be used to signal ATS to not store the response in cache Here is an example: function do_global_read_response() ts.http.set_server_resp_no_store(1) return 0 end
TOP ts.http.set_resp syntax: ts.http.set_resp(CODE, BODY) context: do_remap or do_global_* description: This function can be used to set response for the client with the CODE status and BODY string. Here is an example: function do_remap() ts.http.set_resp(403,"Document access failed :) \n") return 0 end
We will get the response like this:
HTTP/1.1 403 Forbidden Date: Thu, 20 Mar 2014 06:12:43 GMT Connection: close Server: ATS/5.0.0
138 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Cache-Control: no-store Content-Type: text/html Content-Language: en Content-Length: 27
Document access failed :)
TOP ts.http.get_cache_lookup_status syntax: ts.http.get_cache_lookup_status() context: function @ TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE hook point description: This function can be used to get cache lookup status. Here is an example: function cache_lookup() local cache_status= ts.http.get_cache_lookup_status() if cache_status == TS_LUA_CACHE_LOOKUP_HIT_FRESH then print('hit') else print('not hit') end end function do_remap() ts.hook(TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE, cache_lookup) return 0 end
TOP ts.http.set_cache_lookup_status syntax: ts.http.set_cache_lookup_status() context: function after TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE hook point description: This function can be used to set cache lookup status. Here is an example: function cache_lookup() local cache_status= ts.http.get_cache_lookup_status() if cache_status == TS_LUA_CACHE_LOOKUP_HIT_FRESH then print('hit') else print('not hit') end ts.http.set_cache_lookup_status(TS_LUA_CACHE_LOOKUP_MISS) end function do_remap() ts.hook(TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE, cache_lookup)
3.7. Plugins 139 Apache-Trafﬁcserver-Server Documentation, latest
 return 0 end
TOP
Http cache lookup status constants context: global
TS_LUA_CACHE_LOOKUP_MISS (0) TS_LUA_CACHE_LOOKUP_HIT_STALE (1) TS_LUA_CACHE_LOOKUP_HIT_FRESH (2) TS_LUA_CACHE_LOOKUP_SKIPPED (3)
TOP ts.cached_response.get_status syntax: status = ts.cached_response.get_status() context: function @ TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE hook point or later description: This function can be used to retrieve the status code of the cached response. A Lua number will be returned. Here is an example: function cache_lookup() local cache_status= ts.http.get_cache_lookup_status() if cache_status == TS_LUA_CACHE_LOOKUP_HIT_FRESH then code= ts.cached_response.get_status() print(code)-- 200 end end function do_remap() ts.hook(TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE, cache_lookup) return 0 end
TOP ts.cached_response.get_version syntax: ver = ts.cached_response.get_version() context: function @ TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE hook point or later description: Return the http version string of the cached response. Current possible values are 1.0, 1.1, and 0.9. TOP
140 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest ts.cached_response.header.HEADER syntax: VALUE = ts.cached_response.header.HEADER syntax: VALUE = ts.cached_response.header[HEADER] context: function @ TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE hook point or later description: get the current cached response’s HEADER. Here is an example: function cache_lookup() local status= ts.http.get_cache_lookup_status() if status == TS_LUA_CACHE_LOOKUP_HIT_FRESH then local ct= ts.cached_response.header['Content-Type'] print(ct)-- text/plain end end function do_remap() ts.hook(TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE, cache_lookup) return 0 end
TOP ts.cached_response.get_headers syntax: ts.cached_response.get_headers() context: function @ TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE hook point or later description: Returns a Lua table holding all the headers for the current cached response. Here is an example: function cache_lookup() local status= ts.http.get_cache_lookup_status() if status == TS_LUA_CACHE_LOOKUP_HIT_FRESH then hdrs= ts.cached_response.get_headers() for k, v in pairs(hdrs) do print(k..':'..v) end end end function do_remap() ts.hook(TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE, cache_lookup) return 0 end
We will get the output:
Connection: keep-alive Content-Type: text/plain Date: Thu, 20 Mar 2014 06:12:20 GMT Cache-Control: max-age=864000 Last-Modified: Sun, 19 May 2013 13:22:01 GMT
3.7. Plugins 141 Apache-Trafﬁcserver-Server Documentation, latest
Accept-Ranges: bytes Content-Length: 15 Server: ATS/5.0.0
TOP ts.server_request.get_uri syntax: ts.server_request.get_uri() context: function @ TS_LUA_HOOK_SEND_REQUEST_HDR hook point or later description: This function can be used to retrieve the server request’s path. Here is an example: function send_request() local uri= ts.server_request.get_uri() print(uri) end function do_remap() ts.hook(TS_LUA_HOOK_SEND_REQUEST_HDR, send_request) return 0 end
Then GET /am.txt?a=1 will yield the output: /am.txt TOP ts.server_request.set_uri syntax: ts.server_request.set_uri(PATH) context: function @ TS_LUA_HOOK_SEND_REQUEST_HDR hook point description: This function can be used to override the server request’s path. The PATH argument must be a Lua string and starts with / TOP ts.server_request.get_uri_args syntax: ts.server_request.get_uri_args() context: function @ TS_LUA_HOOK_SEND_REQUEST_HDR hook point or later description: This function can be used to retrieve the server request’s query string. Here is an example: function send_request() local query= ts.server_request.get_uri_args() print(query) end
142 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 function do_remap() ts.hook(TS_LUA_HOOK_SEND_REQUEST_HDR, send_request) return 0 end
Then GET /st?a=1&b=2 will yield the output: a=1&b=2 TOP ts.server_request.set_uri_args syntax: ts.server_request.set_uri_args(QUERY_STRING) context: function @ TS_LUA_HOOK_SEND_REQUEST_HDR hook point description: This function can be used to override the server request’s query string. ts.server_request.set_uri_args('n=6&p=7')
TOP ts.server_request.get_uri_params syntax: ts.server_request.get_uri_params() context: function @ TS_LUA_HOOK_SEND_REQUEST_HDR hook point or later description: This function can be used to retrieve the server request’s parameter string. Here is an example: function send_request() local query= ts.server_request.get_uri_params() print(query) end function do_remap() ts.hook(TS_LUA_HOOK_SEND_REQUEST_HDR, send_request) return 0 end
Then GET /st;a=1 will yield the output: a=1 TOP ts.server_request.set_uri_params syntax: ts.server_request.set_uri_params(PARAMETER_STRING) context: function @ TS_LUA_HOOK_SEND_REQUEST_HDR hook point description: This function can be used to override the server request’s parameter string.
3.7. Plugins 143 Apache-Trafﬁcserver-Server Documentation, latest
 ts.server_request.set_uri_params('n=6')
TOP
 ts.server_request.header.HEADER
 syntax: ts.server_request.header.HEADER = VALUE syntax: ts.server_request.header[HEADER] = VALUE syntax: VALUE = ts.server_request.header.HEADER context: function @ TS_LUA_HOOK_SEND_REQUEST_HDR hook point or later description: Set, add to, clear or get the current server request’s HEADER. Here is an example:
 function send_request() local ua= ts.server_request.header['User-Agent'] print(ua) ts.server_request.header['Accept-Encoding']='gzip' end
 function do_remap() ts.hook(TS_LUA_HOOK_SEND_REQUEST_HDR, send_request) return 0 end
Then GET /st HTTP/1.1\r\nHost: b.tb.cn\r\nUser-Agent: Mozilla/5.0\r\n... will yield the output: Mozilla/5.0 TOP ts.server_request.get_headers syntax: ts.server_request.get_headers() context: function @ TS_LUA_HOOK_SEND_REQUEST_HDR hook point or later description: Returns a Lua table holding all the headers for the current server request. Here is an example:
 function send_request() hdrs= ts.cached_response.get_headers() for k, v in pairs(hdrs) do print(k..':'..v) end end
 function do_remap() ts.hook(TS_LUA_HOOK_SEND_REQUEST_HDR, send_request) return 0 end
144 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
We will get the output:
Host: b.tb.cn User-Agent: curl/7.19.7 Accept: */*
TOP ts.server_request.server_addr.get_addr syntax: ts.server_request.server_addr.get_addr() context: do_remap or do_global_* or later description: This function can be used to get socket address of the origin server. The ts.server_request.server_addr.get_addr function returns three values, ip is a string, port and family is number. Here is an example: function do_global_send_request() ip, port, family= ts.server_request.server_addr.get_addr() print(ip)-- 192.168.231.17 print(port)-- 80 print(family)--2(AF_INET) end
TOP ts.server_request.server_addr.get_ip syntax: ts.server_request.server_addr.get_ip() context: do_remap or do_global_* or later description: This function can be used to get ip address of the origin server. The ts.server_request.server_addr.get_ip function returns ip as a string. Here is an example: function do_global_send_request() ip= ts.server_request.server_addr.get_ip() print(ip)-- 192.168.231.17 end
TOP ts.server_request.server_addr.get_port syntax: ts.server_request.server_addr.get_port() context: do_remap or do_global_* or later description: This function can be used to get port of the origin server. The ts.server_request.server_addr.get_port function returns port as number. Here is an example:
3.7. Plugins 145 Apache-Trafﬁcserver-Server Documentation, latest
 function do_global_send_request() port= ts.server_request.server_addr.get_port() print(port)-- 80 end
TOP ts.server_request.server_addr.get_outgoing_port syntax: ts.server_request.server_addr.get_outgoing_port() context: do_remap or do_global_* or later description: This function can be used to get outgoing port to the origin server. The ts.server_request.server_addr.get_outgoing_port function returns outgoing port as number. Here is an example: function do_global_send_request() port= ts.server_request.server_addr.get_outgoing_port() print(port)-- 50880 end
TOP ts.server_request.get_url_host syntax: host = ts.server_request.get_url_host() context: function @ TS_LUA_HOOK_SEND_REQUEST_HDR hook point description: Return the host ﬁeld of the request url. Here is an example: function send_request() local url_host= ts.server_request.get_url_host() print(url_host) end function do_remap() ts.hook(TS_LUA_HOOK_SEND_REQUEST_HDR, send_request) return 0 end
Then GET http://abc.com/p2/a.txt HTTP/1.1 will yield the output: abc.com TOP ts.server_request.set_url_host syntax: ts.server_request.set_url_host(str) context: function @ TS_LUA_HOOK_SEND_REQUEST_HDR hook point
146 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest description: Set host ﬁeld of the request url with str. This function is used to change the host name in the GET request to next tier Here is an example: function send_request() ts.server_request.set_url_host("") ts.server_request.set_url_scheme("") end function do_remap() ts.hook(TS_LUA_HOOK_SEND_REQUEST_HDR, send_request) return 0 end
The GET request like this:
+++++++++ Proxy's Request +++++++++ - State Machine Id: 5593 GET http://origin.com/dir1/a.txt HTTP/1.1 User-Agent: curl/7.29.0 Host: abc.com Accept: / Client-ip: 135.xx.xx.xx X-Forwarded-For: 135.xx.xx.xx
Will be changed to:
+++++++++ Proxy's Request +++++++++ - State Machine Id: 5593 GET /dir1/a.txt HTTP/1.1 User-Agent: curl/7.29.0 Host: abc.com Accept: / Client-ip: 135.xx.xx.xx X-Forwarded-For: 135.xx.xx.xx
TOP ts.server_request.get_url_scheme syntax: scheme = ts.server_request.get_url_scheme() context: function @ TS_LUA_HOOK_SEND_REQUEST_HDR hook point description: Return the scheme ﬁeld of the request url. Here is an example: function send_request() local url_scheme= ts.server_request.get_url_scheme() print(url_host) end function do_remap() ts.hook(TS_LUA_HOOK_SEND_REQUEST_HDR, send_request) return 0 end
3.7. Plugins 147 Apache-Trafﬁcserver-Server Documentation, latest
Then GET /liuyurou.txt HTTP/1.1\r\nHost: 192.168.231.129:8080\r\n... will yield the output: http TOP ts.server_request.set_url_scheme syntax: ts.server_request.set_url_scheme(str) context: function @ TS_LUA_HOOK_SEND_REQUEST_HDR hook point description: Set scheme ﬁeld of the request url with str. This function is used to change the scheme of the server request. TOP ts.server_response.get_status syntax: status = ts.server_response.get_status() context: function @ TS_LUA_HOOK_READ_RESPONSE_HDR hook point or later description: This function can be used to retrieve the status code of the origin server’s response. A Lua number will be returned. Here is an example: function read_response() local code= ts.server_response.get_status() print(code)-- 200 end function do_remap() ts.hook(TS_LUA_HOOK_READ_RESPONSE_HDR, read_response) return 0 end
TOP‘ ts.server_response.set_status syntax: ts.server_response.set_status(NUMBER) context: function @ TS_LUA_HOOK_READ_RESPONSE_HDR hook point description: This function can be used to set the status code of the origin server’s response. Here is an example: function read_response() ts.server_response.set_status(404) end function do_remap() ts.hook(TS_LUA_HOOK_READ_RESPONSE_HDR, read_response)
148 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 return 0 end
TOP‘ ts.server_response.get_version syntax: ver = ts.server_response.get_version() context: function @ TS_LUA_HOOK_READ_RESPONSE_HDR hook point or later. description: Return the http version string of the server response. Current possible values are 1.0, 1.1, and 0.9. TOP ts.server_response.set_version syntax: ts.server_response.set_version(VERSION_STR) context: function @ TS_LUA_HOOK_READ_RESPONSE_HDR hook point description: Set the http version of the server response with the VERSION_STR ts.server_response.set_version('1.0')
TOP ts.server_response.header.HEADER syntax: ts.server_response.header.HEADER = VALUE syntax: ts.server_response.header[HEADER] = VALUE syntax: VALUE = ts.server_response.header.HEADER context: function @ TS_LUA_HOOK_READ_RESPONSE_HDR hook point or later. description: Set, add to, clear or get the current server response’s HEADER. Here is an example: function read_response() local ct= ts.server_response.header['Content-Type'] print(ct) ts.server_response.header['Cache-Control']='max-age=14400' end function do_remap() ts.hook(TS_LUA_HOOK_READ_RESPONSE_HDR, read_response) return 0 end
We will get the output: text/html TOP‘
3.7. Plugins 149 Apache-Trafﬁcserver-Server Documentation, latest ts.server_response.get_headers syntax: ts.server_response.get_headers() context: function @ TS_LUA_HOOK_READ_RESPONSE_HDR hook point or later description: Returns a Lua table holding all the headers for the current server response. Here is an example: function read_response() hdrs= ts.server_response.get_headers() for k, v in pairs(hdrs) do print(k..':'..v) end end function do_remap() ts.hook(TS_LUA_HOOK_READ_RESPONSE_HDR, read_response) return 0 end
We will get the output:
Server: nginx/1.5.9 Date: Tue, 18 Mar 2014 10:12:25 GMT Content-Type: text/html Content-Length: 555 Last-Modified: Mon, 19 Aug 2013 14:25:55 GMT Connection: keep-alive ETag:"52122af3-22b" Cache-Control: max-age=14400 Accept-Ranges: bytes
TOP ts.client_response.get_status syntax: status = ts.client_response.get_status() context: function @ TS_LUA_HOOK_SEND_RESPONSE_HDR hook point description: This function can be used to retrieve the status code of the response to the client. A Lua number will be returned. Here is an example: function send_response() local code= ts.client_response.get_status() print(code)-- 200 end function do_remap() ts.hook(TS_LUA_HOOK_SEND_RESPONSE_HDR, send_response) return 0 end
TOP
150 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest ts.client_response.set_status syntax: ts.client_response.set_status(NUMBER) context: function @ TS_LUA_HOOK_SEND_RESPONSE_HDR hook point description: This function can be used to set the status code of the response to the client. Here is an example: function send_response() ts.client_response.set_status(404) end function do_remap() ts.hook(TS_LUA_HOOK_SEND_RESPONSE_HDR, send_response) return 0 end
TOP ts.client_response.get_version syntax: ver = ts.client_response.get_version() context: function @ TS_LUA_HOOK_SEND_RESPONSE_HDR hook point. description: Return the http version string of the response to the client. Current possible values are 1.0, 1.1, and 0.9. TOP ts.client_response.set_version syntax: ts.client_response.set_version(VERSION_STR) context: function @ TS_LUA_HOOK_SEND_RESPONSE_HDR hook point description: Set the http version of the response to the client with the VERSION_STR ts.client_response.set_version('1.0')
TOP ts.client_response.header.HEADER syntax: ts.client_response.header.HEADER = VALUE syntax: ts.client_response.header[HEADER] = VALUE syntax: VALUE = ts.client_response.header.HEADER context: function @ TS_LUA_HOOK_SEND_RESPONSE_HDR hook point. description: Set, add to, clear or get the current client response’s HEADER. Here is an example:
3.7. Plugins 151 Apache-Trafﬁcserver-Server Documentation, latest
 function send_response() local ct= ts.client_response.header['Content-Type'] print(ct) ts.client_response.header['Cache-Control']='max-age=3600' end function do_remap() ts.hook(TS_LUA_HOOK_SEND_RESPONSE_HDR, send_response) return 0 end
We will get the output: text/html TOP ts.client_response.get_headers syntax: ts.client_response.get_headers() context: function @ TS_LUA_HOOK_SEND_RESPONSE_HDR hook point. description: Returns a Lua table holding all the headers for the current client response. Here is an example: function send_response() hdrs= ts.client_response.get_headers() for k, v in pairs(hdrs) do print(k..':'..v) end end function do_remap() ts.hook(TS_LUA_HOOK_SEND_RESPONSE_HDR, send_response) return 0 end
We will get the output:
Server: ATS/5.0.0 Date: Tue, 18 Mar 2014 10:12:25 GMT Content-Type: text/html Transfer-Encoding: chunked Last-Modified: Mon, 19 Aug 2013 14:25:55 GMT Connection: keep-alive Cache-Control: max-age=14400 Age: 2641 Accept-Ranges: bytes
TOP ts.client_response.set_error_resp syntax: ts.client_response.set_error_resp(CODE, BODY)
152 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest context: function @ TS_LUA_HOOK_SEND_RESPONSE_HDR hook point. description: This function can be used to set the error response to the client. With this function we can jump to send error response to the client if exception exists, meanwhile we should return -1 from the function where exception raises. Here is an example: function send_response() ts.client_response.set_error_resp(404,'bad luck :( \n') end function cache_lookup() return -1 end function do_remap() ts.hook(TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE, cache_lookup) ts.hook(TS_LUA_HOOK_SEND_RESPONSE_HDR, send_response) return 0 end
We will get the response like this:
HTTP/1.1 404 Not Found Date: Tue, 18 Mar 2014 11:16:00 GMT Connection: keep-alive Server: ATS/5.0.0 Content-Length: 12 bad luck :(
TOP
Number constants context: global
TS_LUA_INT64_MAX (9223372036854775808) TS_LUA_INT64_MIN (-9223372036854775808L)
These constants are usually used in transform handler. TOP ts.http.resp_cache_transformed syntax: ts.http.resp_cache_transformed(BOOL) context: do_remap or do_global_* or later description: This function can be used to tell trafﬁcserver whether to cache the transformed data. Here is an example:
3.7. Plugins 153 Apache-Trafﬁcserver-Server Documentation, latest
 function upper_transform(data, eos) return string.upper(data), eos end function do_remap() ts.hook(TS_LUA_RESPONSE_TRANSFORM, upper_transform) ts.http.resp_cache_transformed(0) return 0 end
This function is usually called after we hook TS_LUA_RESPONSE_TRANSFORM. TOP ts.http.resp_cache_untransformed syntax: ts.http.resp_cache_untransformed(BOOL) context: do_remap or do_global_* or later description: This function can be used to tell trafﬁcserver whether to cache the untransformed data. Here is an example: function upper_transform(data, eos) return string.upper(data), eos end function do_remap() ts.hook(TS_LUA_RESPONSE_TRANSFORM, upper_transform) ts.http.resp_cache_untransformed(1) return 0 end
This function is usually called after we hook TS_LUA_RESPONSE_TRANSFORM. TOP ts.http.resp_transform.get_upstream_bytes syntax: ts.http.resp_transform.get_upstream_bytes() context: transform handler description: This function can be used to retrive the total bytes to be received from the upstream. If we got chunked response body from origin server, TS_LUA_INT64_MAX will be returned. Here is an example: local APPEND_DATA='TAIL \n' function append_transform(data, eos) if ts.ctx['len_set'] == nil then local sz= ts.http.resp_transform.get_upstream_bytes() if sz~= TS_LUA_INT64_MAX then ts.http.resp_transform.set_downstream_bytes(sz+ string.len(APPEND_DATA)) end
154 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 ts.ctx['len_set']= true end
 if eos ==1 then return data.. APPEND_DATA, eos else return data, eos end end function do_remap() ts.hook(TS_LUA_RESPONSE_TRANSFORM, append_transform) ts.http.resp_cache_transformed(0) ts.http.resp_cache_untransformed(1) return 0 end
TOP ts.http.resp_transform.set_downstream_bytes syntax: ts.http.resp_transform.set_downstream_bytes(NUMBER) context: transform handler description: This function can be used to set the total bytes to be sent to the downstream. Sometimes we want to set Content-Length header in client_response, and this function should be called before any real data is returned from the transform handler. TOP ts.http.skip_remapping_set syntax: ts.http.skip_remapping_set(BOOL) context: do_global_read_request description: This function can be used to tell trafﬁcserver to skip doing remapping Here is an example: function do_global_read_request() ts.http.skip_remapping_set(1); ts.client_request.header['Host']='www.yahoo.com' return 0 end
This function is usually called in do_global_read_request function TOP ts.http.is_internal_request syntax: ts.http.is_internal_request() context: do_remap or do_global_* or later
3.7. Plugins 155 Apache-Trafﬁcserver-Server Documentation, latest description: This function can be used to tell is a request is internal or not Here is an example: function do_global_read_request() local internal= ts.http.is_internal_request() ts.debug(internal) return 0 end
TOP ts.http.transaction_count syntax: ts.http.transaction_count() context: do_remap or do_global_* or later description: This function returns the number of transaction in this connection Here is an example function do_remap() local count= ts.http.transaction_count() ts.debug(tostring(count)) return 0 end
TOP ts.add_package_path syntax: ts.add_package_path(lua-style-path-str) context: init stage of the lua script description: Adds the Lua module search path used by scripts. The path string is in standard Lua path form. Here is an example: ts.add_package_path('/home/a/test/lua/pac/?.lua') local nt= require("nt") function do_remap() print(nt.t9(7979)) return 0 end
TOP ts.add_package_cpath syntax: ts.add_package_cpath(lua-style-cpath-str) context: init stage of the lua script description: Adds the Lua C-module search path used by scripts.
156 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
The cpath string is in standard Lua cpath form. Here is an example: ts.add_package_cpath('/home/a/test/c/module/?.so') local ma= require("ma") function do_remap() print(ma.ft()) return 0 end
TOP ts.md5 syntax: digest = ts.md5(str) context: global description: Returns the hexadecimal representation of the MD5 digest of the str argument. Here is an example: function do_remap() uri= ts.client_request.get_uri() print(uri)--/foo print(ts.md5(uri))--1effb2475fcfba4f9e8b8a1dbc8f3caf end
TOP ts.md5_bin syntax: digest = ts.md5_bin(str) context: global description: Returns the binary form of the MD5 digest of the str argument. Here is an example: function do_remap() uri= ts.client_request.get_uri() bin= ts.md5_bin(uri) end
TOP ts.sha1 syntax: digest = ts.sha1(str) context: global description: Returns the hexadecimal representation of the SHA-1 digest of the str argument. Here is an example:
3.7. Plugins 157 Apache-Trafﬁcserver-Server Documentation, latest
 function do_remap() uri= ts.client_request.get_uri() print(uri)--/foo print(ts.sha1(uri))--6dbd548cc03e44b8b44b6e68e56255ce4273ae49 end
TOP
 ts.sha1_bin
 syntax: digest = ts.sha1_bin(str) context: global description: Returns the binary form of the SHA-1 digest of the str argument. Here is an example:
 function do_remap() uri= ts.client_request.get_uri() bin= ts.sha1_bin(uri) end
TOP
 ts.base64_encode
 syntax: value = ts.base64_encode(str) context: global description: Returns the base64 encoding of the str argument. Here is an example:
 function do_remap() uri= ts.client_request.get_uri() value= ts.base64_encode(uri) end
TOP
 ts.base64_decode
 syntax: value = ts.base64_decode(str) context: global description: Returns the base64 decoding of the str argument. Here is an example:
 function do_remap() uri= ts.client_request.get_uri() encoded_value= ts.base64_encode(uri) decoded_value= ts.base64_decode(encoded_value) end
158 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TOP
 ts.escape_uri
 syntax: value = ts.escape_uri(str) context: global description: Returns the uri-escaped value of the str argument. Here is an example:
 function do_remap() test='/some value/' value= ts.escape_uri(test) end
TOP
 ts.unescape_uri
 syntax: value = ts.unescape_uri(str) context: global description: Returns the uri-unescaped value of the str argument. Here is an example:
 function do_remap() test='/some value/' escaped_value= ts.escape_uri(test) unescaped_value= ts.unescape_uri(escaped_value) end
TOP
 ts.fetch
 syntax: res = ts.fetch(url, table?) context: after do_remap description: Issues a synchronous but still non-block http request with the url and the optional table. Returns a Lua table with serveral slots (res.status, res.header, res.body, and res.truncated). res.status holds the response status code. res.header holds the response header table. res.body holds the response body which may be truncated, you need to check res.truncated to see if the data is truncated. Here is a basic example:
3.7. Plugins 159 Apache-Trafﬁcserver-Server Documentation, latest
 function post_remap() local url= string.format('http:// %s/foo.txt', ts.ctx['host']) local res= ts.fetch(url) if res.status == 200 then print(res.body) end end function do_remap() local inner= ts.http.is_internal_request() if inner~=0 then return 0 end local host= ts.client_request.header['Host'] ts.ctx['host']= host ts.hook(TS_LUA_HOOK_POST_REMAP, post_remap) end
We can set the optional table with serveral members: header holds the request header table. method holds the request method. The default method is ‘GET’. cliaddr holds the request client address in ip:port form. The default cliaddr is ‘127.0.0.1:33333’ Issuing a post request:
 res= ts.fetch('http://xx.com/foo', {method='POST', body='hello world'})
TOP
 ts.fetch_multi
 syntax: vec = ts.fetch_multi({{url, table?}, {url, table?}, ...}) context: after do_remap Just like ts.fetch, but supports multiple http requests running in parallel. This function will fetch all the urls speciﬁed by the input table and return a table which contain all the results in the same order. Here is an example:
 local vec= ts.fetch_multi({ {'http://xx.com/slayer'}, {'http://xx.com/am', {cliaddr='192.168.1.19:35423'}}, {'http://xx.com/naga', {method='POST', body='hello world'}}, })
 for i=1, #(vec) do print(vec[i].status) end
TOP
160 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest ts.http.intercept syntax: ts.http.intercept(FUNCTION, param1?, param2?, ...) context: do_remap or do_global_* description: Intercepts the client request and processes it in FUNCTION with optional params. We should construct the response for the client request, and the request will not be processed by other modules, like hostdb, cache, origin server... Intercept FUNCTION will be executed in a new lua_thread, so we can delivery optional params from old lua_thread to new lua_thread if needed. Here is an example: require'os' function send_data(dstr) local nt= os.time()..' Zheng. \n' local resp='HTTP/1.0 200 OK \r\n'.. 'Server: ATS/3.2.0\r\n'.. 'Content-Type: text/plain\r\n'.. 'Content-Length:'.. string.format(' %d', string.len(nt))..' \r\n'
˓→.. 'Last-Modified:'.. os.date("%a, %d %b%Y%H:%M:%S GMT", os.
˓→time())..' \r\n'.. 'Connection: keep-alive\r\n'.. 'Cache-Control: max-age=7200\r\n'.. 'Accept-Ranges: bytes\r\n\r\n'.. nt print(dstr) ts.say(resp) end function do_remap() ts.http.intercept(send_data,'hello world') return 0 end
Then we will get the response like this:
HTTP/1.1 200OK Server: ATS/5.0.0 Content-Type: text/plain Content-Length: 18 Last-Modified: Tue, 18 Mar 2014 08:23:12 GMT Cache-Control: max-age=7200 Accept-Ranges: bytes Date: Tue, 18 Mar 2014 12:23:12 GMT Age:0 Connection: keep-alive
1395145392 Zheng.
TOP
3.7. Plugins 161 Apache-Trafﬁcserver-Server Documentation, latest ts.http.server_intercept syntax: ts.http.server_intercept(FUNCTION, param1?, param2?, ...) context: do_remap or do_global_* description: Intercepts the server request and acts as the origin server. Just like ts.http.intercept, but this function will intercept the server request, and we can acts as the origin server in FUNCTION. Here is an example: require'os' function process_combo(host) local url1= string.format('http:// %s/css/1.css', host) local url2= string.format('http:// %s/css/2.css', host) local url3= string.format('http:// %s/css/3.css', host)
 local hdr={ ['Host']= host, ['User-Agent']='blur blur', }
 local ct={ header= hdr, method='GET' }
 local arr= ts.fetch_multi( { {url1, ct}, {url2, ct}, {url3, ct}, })
 local ctype= arr[1].header['Content-Type'] local body= arr[1].body.. arr[2].body.. arr[3].body
 local resp='HTTP/1.1 200 OK \r\n'.. 'Server: ATS/5.2.0\r\n'.. 'Last-Modified:'.. os.date("%a, %d %b%Y%H:%M:%S GMT", os.
˓→time())..' \r\n'.. 'Cache-Control: max-age=7200\r\n'.. 'Accept-Ranges: bytes\r\n'.. 'Content-Type:'.. ctype..' \r\n'.. 'Content-Length:'.. string.format(' %d', string.len(body))..
˓→'\r\n\r\n'.. body
 ts.say(resp) end function do_remap() local inner= ts.http.is_internal_request() if inner~=0 then return 0 end
162 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 local h= ts.client_request.header['Host'] ts.http.server_intercept(process_combo, h) end
TOP ts.say syntax: ts.say(data) context: intercept or server_intercept description: Write response to ATS within intercept or server_intercept. TOP ts.ﬂush syntax: ts.ﬂush() context: intercept or server_intercept description: Flushes the output to ATS within intercept or server_intercept. In synchronous mode, the function will not return until all output data has been written into the system send buffer. Note that using the Lua coroutine mechanism means that this function does not block the ATS event loop even in the synchronous mode. Here is an example: require'os' function send_data() ss='wo ai yu ye hua \n' local resp='HTTP/1.0 200 OK \r\n'.. 'Server: ATS/3.2.0\r\n'.. 'Content-Type: text/plain\r\n'.. 'Content-Length:'.. string.format(' %d',5 *string.len(ss))..' \r\n ˓→'.. 'Last-Modified:'.. os.date("%a, %d %b%Y%H:%M:%S GMT", os.
˓→time())..' \r\n'.. 'Connection: keep-alive\r\n'.. 'Cache-Control: max-age=7200\r\n'.. 'Accept-Ranges: bytes\r\n\r\n' ts.say(resp) for i=1,5 do ts.say(ss) ts.flush() end end function do_remap() ts.http.intercept(send_data) return 0 end
3.7. Plugins 163 Apache-Trafﬁcserver-Server Documentation, latest
We will get the response like this:
HTTP/1.1 200OK Server: ATS/5.0.0 Content-Type: text/plain Content-Length: 80 Last-Modified: Tue, 18 Mar 2014 08:38:29 GMT Cache-Control: max-age=7200 Accept-Ranges: bytes Date: Tue, 18 Mar 2014 12:38:29 GMT Age:0 Connection: keep-alive wo ai yu ye hua wo ai yu ye hua wo ai yu ye hua wo ai yu ye hua wo ai yu ye hua
TOP ts.sleep syntax: ts.sleep(sec) context: after do_remap description: Sleeps for the speciﬁed seconds without blocking. Behind the scene, this method makes use of the ATS event model. Here is an example: function send_response() ts.sleep(3) end function read_response() ts.sleep(3) end function do_remap() ts.hook(TS_LUA_HOOK_READ_RESPONSE_HDR, read_response) ts.hook(TS_LUA_HOOK_SEND_RESPONSE_HDR, send_response) end
TOP ts.schedule syntax: ts.schedule(THREAD_TYPE, sec, FUNCTION, param1?, param2?, ...) context: after do_remap description: Schedule function to be run after speciﬁed seconds without blocking. Behind the scene, this method makes use of the ATS event model. Here is an example:
164 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 function schedule() ts.debug('test schedule starts') end function cache_lookup() ts.debug('cache-lookup') ts.schedule(TS_LUA_THREAD_POOL_NET,0, schedule) return 0 end function do_remap() ts.hook(TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE, cache_lookup) end
TOP ts.http.conﬁg_int_get syntax: val = ts.http.conﬁg_int_get(CONFIG) context: do_remap or do_global_* or later. description: Conﬁguration option which has a int value can be retrieved with this function. val= ts.http.config_int_get(TS_LUA_CONFIG_HTTP_CACHE_HTTP)
TOP ts.http.conﬁg_int_set syntax: ts.http.conﬁg_int_set(CONFIG, NUMBER) context: do_remap or do_global_* or later. description: This function can be used to overwrite the conﬁguration options. Here is an example: function do_remap() ts.http.config_int_set(TS_LUA_CONFIG_HTTP_CACHE_HTTP,0)-- bypass the cache
˓→processor return 0 end
TOP ts.http.conﬁg_ﬂoat_get syntax: val = ts.http.conﬁg_ﬂoat_get(CONFIG) context: do_remap or do_global_* or later. description: Conﬁguration option which has a ﬂoat value can be retrieved with this function. TOP
3.7. Plugins 165 Apache-Trafﬁcserver-Server Documentation, latest ts.http.conﬁg_ﬂoat_set syntax: ts.http.conﬁg_ﬂoat_set(CONFIG, NUMBER) context: do_remap or do_global_* or later. description: This function can be used to overwrite the conﬁguration options. TOP ts.http.conﬁg_string_get syntax: val = ts.http.conﬁg_string_get(CONFIG) context: do_remap or do_global_* or later. description: Conﬁguration option which has a string value can be retrieved with this function. TOP ts.http.conﬁg_string_set syntax: ts.http.conﬁg_string_set(CONFIG, NUMBER) context: do_remap or do_global_* or later. description: This function can be used to overwrite the conﬁguration options. TOP
Http conﬁg constants context: do_remap or do_global_* or later
TS_LUA_CONFIG_URL_REMAP_PRISTINE_HOST_HDR TS_LUA_CONFIG_HTTP_CHUNKING_ENABLED TS_LUA_CONFIG_HTTP_NEGATIVE_CACHING_ENABLED TS_LUA_CONFIG_HTTP_NEGATIVE_CACHING_LIFETIME TS_LUA_CONFIG_HTTP_CACHE_WHEN_TO_REVALIDATE TS_LUA_CONFIG_HTTP_KEEP_ALIVE_ENABLED_IN TS_LUA_CONFIG_HTTP_KEEP_ALIVE_ENABLED_OUT TS_LUA_CONFIG_HTTP_KEEP_ALIVE_POST_OUT TS_LUA_CONFIG_HTTP_SERVER_SESSION_SHARING_MATCH TS_LUA_CONFIG_NET_SOCK_RECV_BUFFER_SIZE_OUT TS_LUA_CONFIG_NET_SOCK_SEND_BUFFER_SIZE_OUT TS_LUA_CONFIG_NET_SOCK_OPTION_FLAG_OUT TS_LUA_CONFIG_HTTP_FORWARD_PROXY_AUTH_TO_PARENT TS_LUA_CONFIG_HTTP_ANONYMIZE_REMOVE_FROM TS_LUA_CONFIG_HTTP_ANONYMIZE_REMOVE_REFERER TS_LUA_CONFIG_HTTP_ANONYMIZE_REMOVE_USER_AGENT TS_LUA_CONFIG_HTTP_ANONYMIZE_REMOVE_COOKIE TS_LUA_CONFIG_HTTP_ANONYMIZE_REMOVE_CLIENT_IP TS_LUA_CONFIG_HTTP_ANONYMIZE_INSERT_CLIENT_IP TS_LUA_CONFIG_HTTP_RESPONSE_SERVER_ENABLED TS_LUA_CONFIG_HTTP_INSERT_SQUID_X_FORWARDED_FOR TS_LUA_CONFIG_HTTP_SERVER_TCP_INIT_CWND TS_LUA_CONFIG_HTTP_SEND_HTTP11_REQUESTS
166 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TS_LUA_CONFIG_HTTP_CACHE_HTTP TS_LUA_CONFIG_HTTP_CACHE_CLUSTER_CACHE_LOCAL TS_LUA_CONFIG_HTTP_CACHE_IGNORE_CLIENT_NO_CACHE TS_LUA_CONFIG_HTTP_CACHE_IGNORE_CLIENT_CC_MAX_AGE TS_LUA_CONFIG_HTTP_CACHE_IMS_ON_CLIENT_NO_CACHE TS_LUA_CONFIG_HTTP_CACHE_IGNORE_SERVER_NO_CACHE TS_LUA_CONFIG_HTTP_CACHE_CACHE_RESPONSES_TO_COOKIES TS_LUA_CONFIG_HTTP_CACHE_IGNORE_AUTHENTICATION TS_LUA_CONFIG_HTTP_CACHE_CACHE_URLS_THAT_LOOK_DYNAMIC TS_LUA_CONFIG_HTTP_CACHE_REQUIRED_HEADERS TS_LUA_CONFIG_HTTP_INSERT_REQUEST_VIA_STR TS_LUA_CONFIG_HTTP_INSERT_RESPONSE_VIA_STR TS_LUA_CONFIG_HTTP_CACHE_HEURISTIC_MIN_LIFETIME TS_LUA_CONFIG_HTTP_CACHE_HEURISTIC_MAX_LIFETIME TS_LUA_CONFIG_HTTP_CACHE_GUARANTEED_MIN_LIFETIME TS_LUA_CONFIG_HTTP_CACHE_GUARANTEED_MAX_LIFETIME TS_LUA_CONFIG_HTTP_CACHE_MAX_STALE_AGE TS_LUA_CONFIG_HTTP_KEEP_ALIVE_NO_ACTIVITY_TIMEOUT_IN TS_LUA_CONFIG_HTTP_KEEP_ALIVE_NO_ACTIVITY_TIMEOUT_OUT TS_LUA_CONFIG_HTTP_TRANSACTION_NO_ACTIVITY_TIMEOUT_IN TS_LUA_CONFIG_HTTP_TRANSACTION_NO_ACTIVITY_TIMEOUT_OUT TS_LUA_CONFIG_HTTP_TRANSACTION_ACTIVE_TIMEOUT_OUT TS_LUA_CONFIG_HTTP_ORIGIN_MAX_CONNECTIONS TS_LUA_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES TS_LUA_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_DEAD_SERVER TS_LUA_CONFIG_HTTP_CONNECT_ATTEMPTS_RR_RETRIES TS_LUA_CONFIG_HTTP_CONNECT_ATTEMPTS_TIMEOUT TS_LUA_CONFIG_HTTP_POST_CONNECT_ATTEMPTS_TIMEOUT TS_LUA_CONFIG_HTTP_DOWN_SERVER_CACHE_TIME TS_LUA_CONFIG_HTTP_DOWN_SERVER_ABORT_THRESHOLD TS_LUA_CONFIG_HTTP_CACHE_FUZZ_TIME TS_LUA_CONFIG_HTTP_CACHE_FUZZ_MIN_TIME TS_LUA_CONFIG_HTTP_DOC_IN_CACHE_SKIP_DNS TS_LUA_CONFIG_HTTP_BACKGROUND_FILL_ACTIVE_TIMEOUT TS_LUA_CONFIG_HTTP_RESPONSE_SERVER_STR TS_LUA_CONFIG_HTTP_CACHE_HEURISTIC_LM_FACTOR TS_LUA_CONFIG_HTTP_CACHE_FUZZ_PROBABILITY TS_LUA_CONFIG_HTTP_BACKGROUND_FILL_COMPLETED_THRESHOLD TS_LUA_CONFIG_NET_SOCK_PACKET_MARK_OUT TS_LUA_CONFIG_NET_SOCK_PACKET_TOS_OUT TS_LUA_CONFIG_HTTP_INSERT_AGE_IN_RESPONSE TS_LUA_CONFIG_HTTP_CHUNKING_SIZE TS_LUA_CONFIG_HTTP_FLOW_CONTROL_ENABLED TS_LUA_CONFIG_HTTP_FLOW_CONTROL_LOW_WATER_MARK TS_LUA_CONFIG_HTTP_FLOW_CONTROL_HIGH_WATER_MARK TS_LUA_CONFIG_HTTP_CACHE_RANGE_LOOKUP TS_LUA_CONFIG_HTTP_NORMALIZE_AE_GZIP TS_LUA_CONFIG_HTTP_DEFAULT_BUFFER_SIZE TS_LUA_CONFIG_HTTP_DEFAULT_BUFFER_WATER_MARK TS_LUA_CONFIG_HTTP_REQUEST_HEADER_MAX_SIZE TS_LUA_CONFIG_HTTP_RESPONSE_HEADER_MAX_SIZE TS_LUA_CONFIG_HTTP_NEGATIVE_REVALIDATING_ENABLED TS_LUA_CONFIG_HTTP_NEGATIVE_REVALIDATING_LIFETIME TS_LUA_CONFIG_HTTP_ACCEPT_ENCODING_FILTER_ENABLED TS_LUA_CONFIG_SSL_HSTS_MAX_AGE TS_LUA_CONFIG_SSL_HSTS_INCLUDE_SUBDOMAINS TS_LUA_CONFIG_HTTP_CACHE_OPEN_READ_RETRY_TIME TS_LUA_CONFIG_HTTP_CACHE_MAX_OPEN_READ_RETRIES
3.7. Plugins 167 Apache-Trafﬁcserver-Server Documentation, latest
TS_LUA_CONFIG_HTTP_CACHE_RANGE_WRITE TS_LUA_CONFIG_HTTP_POST_CHECK_CONTENT_LENGTH_ENABLED TS_LUA_CONFIG_HTTP_GLOBAL_USER_AGENT_HEADER TS_LUA_CONFIG_HTTP_AUTH_SERVER_SESSION_PRIVATE TS_LUA_CONFIG_HTTP_SLOW_LOG_THRESHOLD TS_LUA_CONFIG_HTTP_CACHE_GENERATION TS_LUA_CONFIG_BODY_FACTORY_TEMPLATE_BASE TS_LUA_CONFIG_HTTP_CACHE_OPEN_WRITE_FAIL_ACTION TS_LUA_CONFIG_HTTP_ENABLE_REDIRECTION TS_LUA_CONFIG_HTTP_NUMBER_OF_REDIRECTIONS TS_LUA_CONFIG_HTTP_CACHE_MAX_OPEN_WRITE_RETRIES TS_LUA_CONFIG_LAST_ENTRY
TOP ts.http.timeout_set syntax: ts.http.timeout_set(CONFIG, NUMBER) context: do_remap or do_global_* or later. description: This function can be used to overwrite the timeout settings. Here is an example: function do_remap() ts.http.timeout_set(TS_LUA_TIMEOUT_DNS, 30)-- 30 seconds return 0 end
TOP
Timeout constants context: do_remap or do_global_* or later
TS_LUA_TIMEOUT_ACTIVE TS_LUA_TIMEOUT_DNS TS_LUA_TIMEOUT_CONNECT TS_LUA_TIMEOUT_NO_ACTIVITY
TOP ts.http.client_packet_mark_set syntax: ts.http.client_packet_mark_set(NUMBER) context: do_remap or do_global_* or later. description: This function can be used to set packet mark for client connection. Here is an example:
168 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 function do_remap() ts.http.client_packet_mark_set(0) return 0 end
TOP ts.http.server_packet_mark_set syntax: ts.http.server_packet_mark_set(NUMBER) context: do_remap or do_global_* or later. description: This function can be used to set packet mark for server connection. TOP ts.http.client_packet_tos_set syntax: ts.http.client_packet_tos_set(NUMBER) context: do_remap or do_global_* or later. description: This function can be used to set packet tos for client connection. TOP ts.http.server_packet_tos_set syntax: ts.http.server_packet_tos_set(NUMBER) context: do_remap or do_global_* or later. description: This function can be used to set packet tos for server connection. TOP ts.http.client_packet_dscp_set syntax: ts.http.client_packet_dscp_set(NUMBER) context: do_remap or do_global_* or later. description: This function can be used to set packet dscp for client connection. TOP ts.http.server_packet_dscp_set syntax: ts.http.server_packet_dscp_set(NUMBER) context: do_remap or do_global_* or later. description: This function can be used to set packet dscp for server connection. TOP
3.7. Plugins 169 Apache-Trafﬁcserver-Server Documentation, latest ts.http.enable_redirect syntax: ts.http.enable_redirect(NUMBER) context: do_remap or do_global_* or later. decription: This function can be used to make transaction follow redirect Here is an example: function do_remap() ts.http.enable_redirect(1) return 0 end
TOP ts.http.set_debug syntax: ts.http.set_debug(NUMBER) context: do_remap or do_global_* or later. decription: This function can be used to enable debug log for the transaction Here is an example: function do_remap() ts.http.set_debug(1) return 0 end
TOP ts.http.cntl_get syntax: val = ts.http.cntl_get(CNTL_TYPE) context: do_remap or do_global_* or later. description: This function can be used to retireve the value of control channel. val= ts.http.cntl_get(TS_LUA_HTTP_CNTL_GET_LOGGING_MODE)
TOP ts.http.cntl_set syntax: ts.http.cntl_set(CNTL_TYPE, BOOL) context: do_remap or do_global_* or later. description: This function can be used to set the value of control channel. Here is an example:
170 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 function do_remap() ts.http.cntl_set(TS_LUA_HTTP_CNTL_SET_LOGGING_MODE,0)-- do not log the
˓→request return 0 end
TOP
Http control channel constants context: do_remap or do_global_* or later
TS_LUA_HTTP_CNTL_GET_LOGGING_MODE TS_LUA_HTTP_CNTL_SET_LOGGING_MODE TS_LUA_HTTP_CNTL_GET_INTERCEPT_RETRY_MODE TS_LUA_HTTP_CNTL_SET_INTERCEPT_RETRY_MODE
TOP ts.http.milestone_get syntax: val = ts.http.milestone_get(MILESTONE_TYPE) context: do_remap or do_global_* or later. description: This function can be used to retireve the various milestone times. They are how long the transaction took to traverse portions of the HTTP state machine. Each milestone value is a fractional number of seconds since the beginning of the transaction. val= ts.http.milestone_get(TS_LUA_MILESTONE_SM_START)
TOP
Milestone constants context: do_remap or do_global_* or later
TS_LUA_MILESTONE_UA_BEGIN TS_LUA_MILESTONE_UA_FIRST_READ TS_LUA_MILESTONE_UA_READ_HEADER_DONE TS_LUA_MILESTONE_UA_BEGIN_WRITE TS_LUA_MILESTONE_UA_CLOSE TS_LUA_MILESTONE_SERVER_FIRST_CONNECT TS_LUA_MILESTONE_SERVER_CONNECT TS_LUA_MILESTONE_SERVER_CONNECT_END TS_LUA_MILESTONE_SERVER_BEGIN_WRITE TS_LUA_MILESTONE_SERVER_FIRST_READ TS_LUA_MILESTONE_SERVER_READ_HEADER_DONE TS_LUA_MILESTONE_SERVER_CLOSE TS_LUA_MILESTONE_CACHE_OPEN_READ_BEGIN TS_LUA_MILESTONE_CACHE_OPEN_READ_END TS_LUA_MILESTONE_CACHE_OPEN_WRITE_BEGIN TS_LUA_MILESTONE_CACHE_OPEN_WRITE_END TS_LUA_MILESTONE_DNS_LOOKUP_BEGIN
3.7. Plugins 171 Apache-Trafﬁcserver-Server Documentation, latest
TS_LUA_MILESTONE_DNS_LOOKUP_END TS_LUA_MILESTONE_SM_START TS_LUA_MILESTONE_SM_FINISH TS_LUA_MILESTONE_PLUGIN_ACTIVE TS_LUA_MILESTONE_PLUGIN_TOTAL
TOP ts.mgmt.get_counter syntax: val = ts.mgmt.get_counter(RECORD_NAME) context: do_remap or do_global_* or later. description: This function can be used to retrieve the record value which has a counter type. n= ts.mgmt.get_counter('proxy.process.http.incoming_requests')
TOP ts.mgmt.get_int syntax: val = ts.mgmt.get_int(RECORD_NAME) context: do_remap or do_global_* or later. description: This function can be used to retrieve the record value which has a int type. TOP ts.mgmt.get_ﬂoat syntax: val = ts.mgmt.get_ﬂoat(RECORD_NAME) context: do_remap or do_global_* or later. description: This function can be used to retrieve the record value which has a ﬂoat type. TOP ts.mgmt.get_string syntax: val = ts.mgmt.get_string(RECORD_NAME) context: do_remap or do_global_* or later. description: This function can be used to retrieve the record value which has a string type. name= ts.mgmt.get_string('proxy.config.product_name')
TOP
172 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest ts.stat_create syntax: val = ts.stat_create(STAT_NAME, RECORDDATA_TYPE, PERSISTENT, SYNC) context: global description: This function can be used to create a statistics record given the name, data type, persistent requirement, and sync requirement. A statistics record table will be created with 4 functions to increment, decrement, get and set the value. stat:increment(value) stat:decrement(value) v= stat:get_value() stat:set_value(value)
Here is an example. local test_stat; function __init__(args) test_stat= ts.stat_create("test_stat", TS_LUA_RECORDDATATYPE_INT, TS_LUA_STAT_PERSISTENT, TS_LUA_STAT_SYNC_COUNT) end function do_global_read_request() local value= test_stat:get_value() ts.debug(value) test_stat:increment(1) return 0 end
TOP ts.stat_ﬁnd syntax: val = ts.stat_create(STAT_NAME) context: global description: This function can be used to ﬁnd a statistics record given the name. A statistics record table will be returned with 4 functions to increment, decrement, get and set the value. That is similar to ts.stat_create() TOP
Todo
• ts.cache_xxx • protocol Currently when we use ts_lua as a global plugin, each global hook is using a separate lua state for the same transaction. This can be wasteful. Also the state cannot be reused for the same transaction across the global hooks. The alternative will be to use a TXN_START hook to create a lua state ﬁrst and then add each global hook in the lua script as transac- tion hook instead. But this will have problem down the road when we need to have multiple plugins to work together
3.7. Plugins 173 Apache-Trafﬁcserver-Server Documentation, latest in some proper orderings. In the future, we should consider different approach, such as creating and maintaining the lua state in the ATS core. TOP
Notes on Unit Testing Lua scripts for ATS Lua Plugin
Follow the steps below to use busted framework to run some unit tests on sample scripts and modules • Build and install lua 5.1.5 using the source code from here - http://www.lua.org/ftp/lua-5.1.tar.gz • Build and install luarocks 2.2.2 from here - https://github.com/keplerproject/luarocks/wiki/Download • Run “sudo luarocks install busted” • Run “sudo luarocks install luacov” • “cd trafﬁcserver/plugins/experimental/ts_lua/ci” • Run “busted -c module_test.lua; luacov”. It will produce “luacov.report.out” containing the code coverage for “module.lua” • Run “busted -c script_test.lua; luacov”. It will produce “luacov.report.out” containing the code coverage for “script.lua” Reference for further information • Busted - http://olivinelabs.com/busted/ • Speciﬁcations for asserts/mocks/stubs/etc inside busted framework: https://github.com/Olivine-Labs/luassert/ tree/master/spec • luacov - https://luarocks.org/modules/hisham/luacov TOP
More docs
• https://github.com/portl4t/ts-lua TOP
Webp Transform Plugin
This plugin converts jpeg and png images and transforms them into webp format. All response with content-type ‘image/jpeg’ or ‘image/png’ will go through the transform. Content-type is changed to ‘image/webp’ on successful transformation.
Installation
Add the following line to plugin.config: webp_transform.so
174 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Note
This plugin only supports jpeg and png and requires Magick++ from ImageMagick. Other image formats can easily be supported.
XDebug Plugin
The XDebug plugin allows HTTP clients to debug the operation of the Trafﬁc Server cache using the default X-Debug header. The plugin is triggered by the presence of the X-Debug or the conﬁgured header in the client request. The contents of this header should be the names of the debug headers that are desired in the response. The XDebug plugin will remove the X-Debug header from the client request and inject the desired headers into the client response. XDebug is a global plugin. It is installed by adding it to the plugin.config ﬁle. It currently takes a single, optional conﬁguration option, --header. E.g. –header=ATS-My-Debug This overrides the default X-Debug header name.
Debugging Headers
The XDebug plugin is able to generate the following debugging headers: Via If the Via header is requested, the XDebug plugin sets the :ts:cv:‘proxy.conﬁg.http.insert_response_via_str‘ conﬁguration variable to 3 for the request. Diags If the Diags header is requested, the XDebug plugin enables the transaction speciﬁc diagnostics for the trans- action. This also requires that :ts:cv:‘proxy.conﬁg.diags.debug.enabled‘ is set to 1. X-Cache-Key The X-Cache-Key header contains the URL that identiﬁes the HTTP object in the Trafﬁc Server cache. This header is particularly useful if a custom cache key is being used. X-Cache The X-Cache header contains the results of any cache lookup. Value Description none No cache lookup was attempted. miss The object was not found in the cache. hit-stale The object was found in the cache, but it was stale. hit-fresh The object was fresh in the cache. skipped The cache lookup was skipped. X-Cache-Generation The cache generation ID for this transaction, as speciﬁed by the :ts:cv:‘proxy.conﬁg.http.cache.generation‘ conﬁguration variable. X-Milestones The X-Milestones header contains detailed information about how long the transaction took to traverse portions of the HTTP state machine. The timing information is obtained from the TSHttpTxnMilestoneGet() API. Each milestone value is a fractional number of seconds since the be- ginning of the transaction.
Collapsed Forwarding Plugin
This is a plugin for Apache Trafﬁc Server that allows to achieve effective connection collapse by blocking all but one of the multiple concurrent requests for the same object from going to the Origin.
3.7. Plugins 175 Apache-Trafﬁcserver-Server Documentation, latest
Installation
To make this plugin available, you must either enable experimental plugins when building Trafﬁc Server:
./configure--enable-experimental-plugins
Or use tsxs to compile the plugin against your current Trafﬁc Server build. To do this, you must ensure that: 1. Development packages for Trafﬁc Server are installed. 2. The tsxs binary is in your path. 3. The version of this plugin you are building, and the version of Trafﬁc Server against which you are building it are compatible. Once those conditions are satisﬁed, enter the source directory for the plugin and perform the following: make-f Makefile.tsxs make-f Makefile.tsxs install
Using the plugin
This plugin functions as a per remap plugin, and it takes two optional arguments for specifying the delay between successive retries and a max number of retries. To activate the plugin in per remap mode, in remap.config, simply append the below to the speciﬁc remap line:
@plugin=collapsed_forwarding.so @pparam=--delay= @pparam=--retries=
Functionality
Trafﬁc Server plugin to allow collapsed forwarding of concurrent requests for the same object. This plugin is based on open_write_fail_action feature, which detects cache open write failure on a cache miss and returns a 502 error along with a special @-header indicating the reason for 502 error. The plugin acts on the error by using an internal redirect follow back to itself, essentially blocking the request until a response arrives, at which point, relies on read-while- writer feature to start downloading the object to all waiting clients. The following conﬁg parameters are assumed to be set for this plugin to work:
:ts:cv:`proxy.config.http.cache.open_write_fail_action` 1 :ts:cv:`proxy.config.cache.enable_read_while_writer` 1 :ts:cv:`proxy.config.http.redirection_enabled` 1 :ts:cv:`proxy.config.http.number_of_redirections` 10 :ts:cv:`proxy.config.http.redirect_use_orig_cache_key` 1 :ts:cv:`proxy.config.http.background_fill_active_timeout` 0 :ts:cv:`proxy.config.http.background_fill_completed_threshold` 0
Additionally, given that collapsed forwarding works based on cache write lock failure detection, the plugin requires cache to be enabled and ready. On a restart, Trafﬁc Server typically takes a few seconds to initialize the cache depending on the cache size and number of dirents. While the cache is not ready yet, collapsed forwarding can not detect the write lock contention and so can not work. The setting :ts:cv:‘proxy.conﬁg.http.wait_for_cache‘ may be enabled which allows blocking incoming connections from being accepted until cache is ready.
176 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
Trafﬁc Server has been affected severely by the Thundering Herd problem caused by its inability to do effective connection collapse of multiple concurrent requests for the same segment. This is especially critical when Trafﬁc Server is used as a solution to use cases such as delivering a large scale video live streaming. This problem results in a speciﬁc behavior where multiple number of requests for the same ﬁle are leaked upstream to the Origin layer choking the upstream bandwidth due to the duplicated large ﬁle downloads or process intensive ﬁle at the Origin layer. This ultimately can cause stability problems on the origin layer disrupting the overall network performance. Trafﬁc Server supports several kind of connection collapse mechanisms including Read-While-Writer (RWW), Stale- While-Revalidate (SWR) etc each very effective dealing with a majority of the use cases that can result in the Thun- dering herd problem. For a large scale Video Streaming scenario, there’s a combination of a large number of revalidations (e.g. media playlists) and cache misses (e.g. media segments) that occur for the same ﬁle. Trafﬁc Server’s RWW works great in collapsing the concurrent requests in such a scenario, however, as described in _admin-configuration-reducing-origin-requests, Trafﬁc Server’s implementation of RWW has a signiﬁcant limitation, which restricts its ability to invoke RWW only when the response headers are already received. This means that any number of concurrent requests for the same ﬁle that are received before the response headers arrive are leaked upstream, which can result in a severe Thundering herd problem, depending on the network latencies (which impact the TTFB for the response headers) at a given instant of time. To address this limitation, Trafﬁc Server supports a few Cache tuning solutions, such as Open Read Retry, and a new feature called Open Write Fail action from 6.0. To understand how these approaches work, it is important to understand the high level ﬂow of how Trafﬁc Server handles a GET request. On receiving a HTTP GET request, Trafﬁc Server generates the cache key (basically, a hash of the request URL) and looks up for the directory entry (dirent) using the generated index. On a cache miss, the lookup fails and Trafﬁc Server then tries to just get a write lock for the cache object and proceeds to the origin to download the object. On the Other hand, if the lookup is successful, meaning, the dirent exists for the generated cache key, Trafﬁc Server tries to obtain a read lock on the cache object to be able to serve it from the cache. If the read lock is not successful (possibly, due to the fact that the object’s being written to at that same instant and the response headers are not in the cache yet), Trafﬁc Server then moves to the next step of trying to obtain an exclusive write lock. If the write lock is already held exclusively by another request (transaction), the attempt fails and at this point Trafﬁc Server simply disables the cache on that transaction and downloads the object in a proxy-only mode:
1). Cache Lookup (lookup for the dirent using the request URL as cache key). 1.1). If lookup fails (cache miss), goto (3). 1.2). If lookup succeeds, try to obtain a read lock, goto (2). 2). Open Cache Read (try to obtain read lock) 2.1). If read lock succeeds, serve from cache, goto (4). 2.2). If read lock fails, goto (3). 3). Open Cache Write (try to obtain write lock). 3.1). If write lock succeeds, download the object into cache and to the client in
˓→parallel 3.2). If write lock fails, disable cache, and download to the client in a proxy-
˓→only mode. 4). Done
As can be seen above, if a majority of concurrent requests arrive before response headers are received, they hit (2.2) and (3.2) above. Open Read Retry can help to repeat (2) after a conﬁgured delay on 2.2, thereby increasing the chances for obtaining a read lock and being able to serve from the cache. However, the Open Read Retry can not help with the concurrent requests that hit (1.1) above, jumping to (3) directly. Only one such request will be able to obtain the exclusive write lock and all other requests are leaked upstream. This is where, the recently developed Trafﬁc Server feature Open Write Fail Action will help. The feature detects the write lock failure and can return a stale copy for a Cache Revalidation or a 5xx status code for a Cache Miss with a special
3.7. Plugins 177 Apache-Trafﬁcserver-Server Documentation, latest internal header <@Ats-Internal> that allows a TS plugin to take other special actions depending on the use-case. collapsed_forwarding plugin catches that error in SEND_RESPONSE_HDR_HOOK and performs an internal 3xx Redirect back to the same host, the conﬁgured number of times with the conﬁgured amount of delay between consecutive retries, allowing to be able to initiate RWW, whenever the response headers are received for the request that was allowed to go to the Origin. More details are available at https://docs.trafﬁcserver.apache.org/en/6.0.x/admin/http-proxy-caching.en.html#reducing-origin-server-requests-avoiding-the-thundering-herd AuthProxy Delegates the authorization decision of a request to an external HTTP service. AWS S3 Authentication Support for Amazon S3 authentication features. Background Fetch Proactively fetch content from Origin in a way that it will ﬁll the object into cache. Balancer Balances requests across multiple origin servers. Buffer Upload Buffers POST data before connecting to the Origin server. Cache Key Manipulation Allows some common cache key manipulations based on various HTTP request elements. Cache Promote Provides additional control over when an object should be allowed into the cache. Combo Handler Provides an intelligent way to combine multiple URLs into a single URL, and have Apache Trafﬁc Server combine the components into one response. Epic Emits Trafﬁc Server metrics in a format that is consumed by the Epic Network Monitoring System. Escalate Escalate: when the origin returns speciﬁc status codes, retry the request at a secondary origin (failover/fail- action) ESI Implements the Edge Side Includes (ESI) speciﬁcation. Generator Generate arbitrary response data. GeoIP ACL Deny or allow requests based on the source IP geo-location. HIPES Adds support for HTTP Pipes. Memcache Implements the memcache protocol for cache contents. Metalink Implements the Metalink download description format in order to try not to download the same ﬁle twice. MP4 MP4 streaming media. MySQL Remap Allows dynamic remaps from a MySQL database. Signed URLs Adds support for verifying URL signatures for incoming requests to either deny or redirect access. SSL Headers Populate request headers with SSL session information. Stale While Revalidate Refresh content asynchronously while serving stale data. TS Lua Allows plugins to be written in Lua instead of C code. WebP Transform Converts jpeg and png images to webp format. XDebug Allows HTTP clients to debug the operation of the Trafﬁc Server cache using the X-Debug header. Collapsed-Forwarding Allows to Collapse multiple Concurrent requests by downloading once from the Origin and serving all clients in parallel.
178 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Event and Error Monitoring
Logging
Trafﬁc Server generates log ﬁles that contain information about every request it receives and every error it detects. This chapter will examine the various log features, the conﬁguration formats and also examine the various pre-deﬁned log formats that are available.
Understanding Trafﬁc Server Log Files
Trafﬁc Server records information about every transaction (or request) it processes and every error it detects in log ﬁles. Trafﬁc Server keeps three types of log ﬁles: • Error log ﬁles record information about why a particular transaction was in error. • Event log ﬁles (also called access log ﬁles) record information about the state of each transaction Trafﬁc Server processes. • System log ﬁles record system information, including messages about the state of Trafﬁc Server and er- rors/warnings it produces. This kind of information might include a note that event log ﬁles were rolled, a warning that cluster communication timed out, or an error indicating that Trafﬁc Server was restarted. All system information messages are logged with the system-wide logging facility syslog under the dae- mon facility. The syslog.conf(5) conﬁguration ﬁle (stored in the /etc directory) speciﬁes where these messages are logged. A typical location is /var/log/messages (Linux). The syslog(8) process works on a system-wide basis, so it serves as the single repository for messages from all Trafﬁc Server processes (including traffic_server, traffic_manager, and traffic_cop). System information logs observe a static format. Each log entry in the log contains information about the date and time the error was logged, the hostname of the Trafﬁc Server that reported the error, and a description of the error or warning. Refer to Error Messages for a list of the messages logged by Trafﬁc Server. By default, Trafﬁc Server creates both error and event log ﬁles and records system information in sys- tem log ﬁles. You can disable event logging and/or error logging by setting the conﬁguration variable :ts:cv:‘proxy.conﬁg.log.logging_enabled‘ in records.config to one of the following values: Value Description 0 Disable both event and error logging. 1 Enable error logging only. 2 Enable event logging only. 3 Enable both event and error logging. By analyzing the log ﬁles, you can determine how many people use the Trafﬁc Server cache, how much information each person requested, what pages are most popular, and so on. Trafﬁc Server supports several standard log ﬁle formats, such as Squid and Netscape, as well as user-deﬁned custom formats. You can analyze the standard format log ﬁles with off-the-shelf analysis packages. To help with log ﬁle analysis, you can separate log ﬁles so they contain information speciﬁc to protocol or hosts. You can also conﬁgure Trafﬁc Server to roll log ﬁles automatically at speciﬁc intervals during the day or when they reach a certain size.
Managing Logs
Trafﬁc Server enables you to control where event log ﬁles are located and how much space they can consume. Addi- tionally, you can specify how to handle low disk space in the logging directory.
3.8. Event and Error Monitoring 179 Apache-Trafﬁcserver-Server Documentation, latest
Choosing the Logging Directory
By default, Trafﬁc Server writes all event log ﬁles in the logs directory located in the directory where you in- stalled Trafﬁc Server. To change this location, adjust the value of :ts:cv:‘proxy.conﬁg.log.logﬁle_dir‘ in records. config. You will need to either restart Trafﬁc Server or run the command traffic_line -x for changes to take effect.
Controlling Logging Space
Trafﬁc Server enables you to control the amount of disk space that the logging directory can consume. This allows the system to operate smoothly within a speciﬁed space window for a long period of time. After you establish a space limit, Trafﬁc Server continues to monitor the space in the logging directory. When the free space dwindles to the headroom limit, it enters a low space state and takes the following actions: • If the autodelete option (discussed in Rolling Logs) is enabled, then Trafﬁc Server identiﬁes previously-rolled log ﬁles (log ﬁles with the .old extension). It starts deleting ﬁles one by one, beginning with the oldest ﬁle, until it emerges from the low state. Trafﬁc Server logs a record of all deleted ﬁles in the system error log. • If the autodelete option is disabled or there are not enough old log ﬁles to delete for the system to emerge from its low space state, then Trafﬁc Server issues a warning and continues logging until space is exhausted. When available space is consumed, event logging stops. Trafﬁc Server resumes event logging when enough space becomes available for it to exit the low space state. To make space available, either explicitly increase the logging space limit or remove ﬁles from the logging directory manually. You can run a cron(8) script in conjunction with Trafﬁc Server to automatically remove old log ﬁles from the logging directory before Trafﬁc Server enters the low space state. Relocate the old log ﬁles to a temporary partition, where you can run a variety of log analysis scripts. Following analysis, either compress the logs and move to an archive location, or simply delete them.
Setting Log File Management Options
To set log management options, follow the steps below: 1. In the records.config ﬁle, edit the following variables • :ts:cv:‘proxy.conﬁg.log.max_space_mb_for_logs‘ • :ts:cv:‘proxy.conﬁg.log.max_space_mb_headroom‘ 2. Run the command traffic_line -x to apply the conﬁguration changes.
Rolling Logs
Trafﬁc Server provides automatic log ﬁle rolling. At speciﬁc intervals during the day or when log ﬁles reach a certain size, Trafﬁc Server closes its current set of log ﬁles and opens new log ﬁles. Depending on the amount of trafﬁc your servers are exposed to, you may ﬁnd that increasing the frequency of log rolling is beneﬁcial, or even necessary, to maintain manageable log ﬁle sets. Trafﬁc Server nodes processing moderately high levels of trafﬁc may want to start by rolling logs every six hours, and adjusting from there. Log ﬁle rolling offers the following beneﬁts: • It deﬁnes an consistent interval over which log analysis can be performed. • It keeps any single log ﬁle from becoming too large and helps to keep the logging system within the speciﬁed space limits.
180 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
• It provides an easy way to identify ﬁles that are no longer being used so that an automated script can clean the logging directory and run log analysis programs.
Rolled Log Filename Format
Trafﬁc Server provides a consistent naming scheme for rolled log ﬁles that enables you to easily identify log ﬁles. When Trafﬁc Server rolls a log ﬁle, it saves and closes the old ﬁle before it starts a new ﬁle. Trafﬁc Server renames the old ﬁle to include the following information: • The format of the ﬁle (such as squid.log). • The hostname of the Trafﬁc Server that generated the log ﬁle. • Two timestamps separated by a hyphen (-). The ﬁrst timestamp is a lower bound for the timestamp of the ﬁrst record in the log ﬁle. The lower bound is the time when the new buffer for log records is created. Under low load, the ﬁrst timestamp in the ﬁlename can be different from the timestamp of the ﬁrst entry. Under normal load, the ﬁrst timestamp in the ﬁlename and the timestamp of the ﬁrst entry are similar. The second timestamp is an upper bound for the timestamp of the last record in the log ﬁle (this is normally the rolling time). • The sufﬁx .old, which makes it easy for automated scripts to ﬁnd rolled log ﬁles. Timestamps have the following format:
%Y%M%D.%Hh%Mm%Ss-%Y%M%D.%Hh%Mm%Ss
The following table describes the format: Format Description Sample %Y The year in four-digit format. 2000 %M The month in two-digit format, from 01-12. 07 %D The day in two-digit format, from 01-31. 19 %H The hour in two-digit format, from 00-23. 21 %M The minute in two-digit format, from 00-59. 52 %S The second in two-digit format, from 00-59. 36 The following is an example of a rolled log ﬁlename: squid.log.mymachine.20110912.12h00m00s-20000913.12h00m00s.old
The logging system buffers log records before writing them to disk. When a log ﬁle is rolled, the log buffer might be partially full. If it is, then the ﬁrst entry in the new log ﬁle will have a timestamp earlier than the time of rolling. When the new log ﬁle is rolled, its ﬁrst timestamp will be a lower bound for the timestamp of the ﬁrst entry. For example, suppose logs are rolled every three hours, and the ﬁrst rolled log ﬁle is: squid.log.mymachine.20110912.12h00m00s-19980912.03h00m00s.old
If the lower bound for the ﬁrst entry in the log buffer at 3:00:00 is 2:59:47, then the next log ﬁle will have the following timestamp when rolled: squid.log.mymachine.20110912.02h59m47s-19980912.06h00m00s.old
The contents of a log ﬁle are always between the two timestamps. Log ﬁles do not contain overlapping entries, even if successive timestamps appear to overlap.
3.8. Event and Error Monitoring 181 Apache-Trafﬁcserver-Server Documentation, latest
Rolling Intervals
Log ﬁles are rolled at speciﬁc intervals relative to a given hour of the day. Three options may be used to control when log ﬁles are rolled: • A ﬁle size threshold, which will prevent any individual log from growing too large. • The offset hour, which is an hour between 0 (midnight) and 23. • The rolling interval. Both the offset hour and the rolling interval determine when log ﬁle rolling starts. Rolling occurs every rolling interval and at the offset hour. For example, if the rolling interval is six hours and the offset hour is 0 (midnight), then the logs will roll at midnight (00:00), 06:00, 12:00, and 18:00 each day. If the rolling interval is 12 hours and the offset hour is 3, then logs will roll at 03:00 and 15:00 each day. To set log ﬁle rolling options and/or conﬁgure Trafﬁc Server to roll log ﬁles when they reach a certain size, adjust the following settings in records.config: 1. Enable log rolling with :ts:cv:‘proxy.conﬁg.log.rolling_enabled‘.
CONFIG proxy.config.log.rolling_enabled INT1
2. Conﬁgure the upper limit on log ﬁle size with :ts:cv:‘proxy.conﬁg.log.rolling_size_mb‘.
CONFIG proxy.config.log.rolling_size_mb INT 1024
3. Set the offset hour with :ts:cv:‘proxy.conﬁg.log.rolling_offset_hr‘.
CONFIG proxy.config.log.rolling_offset_hr INT0
4. Set the interval (in seconds) with :ts:cv:‘proxy.conﬁg.log.rolling_interval_sec‘.
CONFIG proxy.config.log.rolling_interval_sec INT 21600
5. Run the command traffic_line -x to apply the conﬁguration changes. You can ﬁne-tune log ﬁle rolling settings for a custom log ﬁle in the LogObject speciﬁcation in logs_xml.config. The custom log ﬁle uses the rolling settings in its LogObject, which override the default settings you specify in Trafﬁc Manager or records.config described above.
Separating Logs by Origin
The default log_hosts.config ﬁle is located in the Trafﬁc Server config directory. To record HTTP transac- tions for different origin servers in separate log ﬁles, you must specify the hostname of each origin server on a separate line in log_hosts.config. For example, if you specify the keyword sports, then Trafﬁc Server records all HTTP transactions from sports.yahoo.com and www.foxsports.com in a log ﬁle called squid-sports. log (if the Squid format is enabled).
: If Trafﬁc Server is clustered and you enable log ﬁle collation, then you should use the same log_hosts.config ﬁle on every Trafﬁc Server node in the cluster.
To edit the log hosts list: 1. Enter the hostname of each origin server on a separate line in log_hosts.config.
182 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 webserver1 webserver2 webserver3
2. Run the command traffic_ctl config reload to apply the changes.
Log Formats
This document provides a reference for all the different logging formats Trafﬁc Server supports. Rather than just reading about those formats, you may also want to try our online event log builder for an interactive way of building and understanding log formats.
Binary or ASCII?
You can conﬁgure Trafﬁc Server to create event log ﬁles in either of the following: ASCII These ﬁles are human-readable and can be processed using standard, off-the-shelf log analysis tools. However, Trafﬁc Server must perform additional processing to create the ﬁles in ASCII, which mildly impacts system overhead. ASCII ﬁles also tend to be larger than the equivalent binary ﬁles. By default, ASCII log ﬁles have a .log ﬁlename extension. Binary These ﬁles generate lower system overhead and generally occupy less space on the disk than ASCII ﬁles (depending on the type of information being logged). However, you must use a converter application before you can read or analyze binary ﬁles via standard tools. By default, binary log ﬁles use a .blog ﬁlename extension. While binary log ﬁles typically require less disk space, there are exceptions. For example, the value 0 (zero) requires only one byte to store in ASCII, but requires four bytes when stored as a binary integer. Conversely, if you deﬁne a custom format that logs IP addresses, then a binary log ﬁle would only require four bytes of storage per 32-bit address. However, the same IP address stored in dot notation would require around 15 characters (bytes) in an ASCII log ﬁle. It is wise to consider the type of data that will be logged before you select ASCII or binary for your log ﬁles, if your decision is being driven by storage space concerns. For example, you might try logging to both formats simultaneously for a representative period of time and compare the storage requirements of the two logs to determine whether one or the other provides any measurable savings.
Deﬁning Log Objects
To perform any logging at all on your Trafﬁc Server nodes, you must have at least one LogObject deﬁned in logs_xml.config. These deﬁnitions conﬁgure what logs will be created, the format they will use (covered in the sections Standard Formats and Custom Formats), any ﬁlters which may be applied to events before logging them, and when or how the logs will be rolled.
Log Filters
LogFilter objects, conﬁgured in logs_xml.config allow you to create ﬁlters, which may be applied to LogObject deﬁnitions, limiting the types of entries which will be included in the log output. This may be useful if your Trafﬁc Server nodes receive many events which you have no need to log or analyze.
3.8. Event and Error Monitoring 183 Apache-Trafﬁcserver-Server Documentation, latest
Standard Formats
The standard log formats include Squid, Netscape Common, Netscape extended, and Netscape Extended-2. The standard log ﬁle formats can be analyzed with a wide variety of off-the-shelf log-analysis packages. You should use one of the standard event log formats unless you need information that these formats do not provide. These formats may be used by enabling the :ts:cv:‘proxy.conﬁg.log.custom_logs_enabled‘ setting in records. config and adding appropriate entries to logs_xml.config.
Squid
The following ﬁgure shows a sample log entry in a Squid log ﬁle.
3.11: Sample log entry in squid.log
Field Sym- Description bol 1 cqtq The client request timestamp in Squid format. The time of the client request in seconds since January 1, 1970 UTC (with millisecond resolution). 2 ttms The time Trafﬁc Server spent processing the client request. The number of milliseconds between the time the client established the connection with Trafﬁc Server and the time Trafﬁc Server sent the last byte of the response back to the client. 3 chi The IP address of the client’s host machine. 4 crc/psscThe cache result code; how the cache responded to the request: HIT, MISS, and so on. Cache result codes are described in Cache Result Codes. The proxy response status code (HTTP response status code from Trafﬁc Server to client). 5 psql The length of the Trafﬁc Server response to the client in bytes, including headers and content. 6 cqhm The client request method: GET, POST, and so on. 7 cauc The client request canonical URL; blanks and other characters that might not be parsed by log analysis tools are replaced by escape sequences. The escape sequence is a percentage sign followed by the ASCII code number of the replaced character in hex. 8 caun The username of the authenticated client. A hyphen (-) means that no authentication was required. 9 phr/pqsnThe proxy hierarchy route. The route Trafﬁc Server used to retrieve the object. 10 psct The proxy response content type. The object content type taken from the Trafﬁc Server response header.
Netscape Common
The following ﬁgure shows a sample log entry in a Netscape Common log ﬁle.
184 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
3.12: Sample log entry in common.log
Field Symbol Description 1 chi The IP address of the client’s host machine. 2 – This hyphen (-) is always present in Netscape log entries. 3 caun The authenticated client username. A hyphen (-) means no authentication was required. 4 cqtd The date and time of the client request, enclosed in brackets. 5 cqtx The request line, enclosed in quotes. 6 pssc The proxy response status code (HTTP reply code). 7 pscl The length of the Trafﬁc Server response to the client in bytes.
Netscape Extended
The following ﬁgure shows a sample log entry in a Netscape Extended log ﬁle.
3.13: Sample log entry in extended.log
In addition to ﬁeld 1-7 from the Netscape Common log format above, the Extended format also adds the following ﬁelds: Field Sym- Description bol 8 sssc The origin server response status code. 9 sshl The server response transfer length; the body length in the origin server response to Trafﬁc Server, in bytes. 10 cqbl The client request transfer length; the body length in the client request to Trafﬁc Server, in bytes. 11 pqbl The proxy request transfer length; the body length in the Trafﬁc Server request to the origin server. 12 cqhl The client request header length; the header length in the client request to Trafﬁc Server. 13 pshl The proxy response header length; the header length in the Trafﬁc Server response to the client. 14 pqhl The proxy request header length; the header length in Trafﬁc Server request to the origin server. 15 sshl The server response header length; the header length in the origin server response to Trafﬁc Server. 16 tts The time Trafﬁc Server spent processing the client request; the number of seconds between the time that the client established the connection with Trafﬁc Server and the time that Trafﬁc Server sent the last byte of the response back to the client.
3.8. Event and Error Monitoring 185 Apache-Trafﬁcserver-Server Documentation, latest
Netscape Extended2
The following ﬁgure shows a sample log entry in a Netscape Extended2 log ﬁle.
3.14: Sample log entry in extended2.log
In addition to ﬁeld 1-16 from the log formats above, the Extended2 format also adds the following ﬁelds: Field Sym- Description bol 17 phr The proxy hierarchy route; the route Trafﬁc Server used to retrieve the object. 18 cfsc The client ﬁnish status code: FIN if the client request completed successfully or INTR if the client request was interrupted. 19 pfsc The proxy ﬁnish status code: FIN if the Trafﬁc Server request to the origin server completed successfully or INTR if the request was interrupted. 20 crc The cache result code; how the Trafﬁc Server cache responded to the request: HIT, MISS, and so on. Cache result codes are listed in Cache Result Codes.
Custom Formats
Deﬁning a Format
Custom logging formats in Trafﬁc Server are deﬁned by editing logs_xml.config and adding new LogFormat entries for each format you wish to deﬁne. The syntax is fairly simple: every LogFormat element should contain at least two child elements (additional elements are used for features such as log summarization and are covered elsewhere): •A  which contains an arbitrary string (using only the allowed characters: [a-z0-9]) naming your custom format. •A  which deﬁnes the ﬁelds that will populate each entry in the custom logs, as well as the order in which they appear. A very simple example format, which contains only the timestamp of when the event began and the canonical URL of the request, and named myformat would be written as follows:
  %"/> 
You may include as many custom ﬁeld codes as you wish. The full list of codes available can be found in Custom Logging Fields. You may also include any literal characters in your format. For example, if we wished to separate the timestamp and canonical URL in our customer format above with a slash instead of a space, or even a slash surrounded by spaces, we could do so by just adding the desired characters to the format string:
186 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
%/%
You may deﬁne as many custom formats as you wish. To apply changes to custom formats, you will need to run the command traffic_line -x after saving your changes to logs_xml.config.
Custom Logging Fields
The following list describes Trafﬁc Server custom logging ﬁelds. {HTTP header field name}cqh Logs the information in the requested ﬁeld of the client request HTTP header. For example, %<{Accept-Language}cqh> logs the Accept-Language: ﬁeld in client request headers.
: ecqh is the URL-encoded version of this map
{HTTP header field name}pqh Logs the information in the requested ﬁeld of the proxy request HTTP header. For example, %<{Authorization}pqh> logs the Authorization: ﬁeld in proxy request head- ers.
: epqh is the URL-encoded version of this map
{HTTP header field name}psh Logs the information in the requested ﬁeld of the proxy response HTTP header. For example, %<{Retry-After}psh> logs the Retry-After: ﬁeld in proxy response headers.
: epsh is the URL-encoded version of this map
{HTTP header field name}ssh Logs the information in the requested ﬁeld of the server response HTTP header. For example, %<{Age}ssh> logs the Age: ﬁeld in server response headers.
: essh is the URL-encoded version of this map
{HTTP header field name}cssh Logs the information in the requested ﬁeld of the cached server response HTTP header. For example, %<{Age}cssh> logs the Age: ﬁeld in the cached server response headers.
: ecssh is the URL-encoded version of this map
 caun The client authenticated username; result of the RFC931/ident lookup of the client username. cfsc The client ﬁnish status code; speciﬁes whether the client request to Trafﬁc Server was successfully completed (FIN) or interrupted (INTR). chi The IP address of the client’s host machine. chih The IP address of the client’s host machine in hexadecimal. hii This is the incoming (interface) IP address for Trafﬁc Server, in otherwords this is the IP address the client connected to. hiih The the incoming (interface) IP address in hexadecimal. chp The port number of the client’s host machine.
3.8. Event and Error Monitoring 187 Apache-Trafﬁcserver-Server Documentation, latest cqbl The client request transfer length; the body length in the client request to Trafﬁc Server (in bytes). cqhl The client request header length; the header length in the client request to Trafﬁc Server. cqhm The HTTP method in the client request to Trafﬁc Server: GET, POST, and so on (subset of cqtx). cqhv The client request HTTP version. cqpv The client request protocol and version. cqtd The client request timestamp. Speciﬁes the date of the client request in the format yyyy-mm-dd, where yyyy is the 4-digit year, mm is the 2-digit month, and dd is the 2-digit day. cqtn The client request timestamp; date and time of the client’s request (in the Netscape timestamp format). cqtq The time of the client request since January 1, 1970 UTC (epoch), with millisecond resolution. cqts The time of the client request since January 1, 1970 UTC (EPOCH), with second resolution. cqtt The client request timestamp. The time of the client request in the format hh:mm:ss, where hh is the two-digit hour in 24-hour format, mm is the two-digit minutes value, and ss is the 2-digit seconds value (for example, 16:01:19). cqtr The TCP reused status; indicates if this client request went through an already established connection. cqssl The SSL client request status indicates if this client connection is over SSL. cqssr The SSL session ticket reused status; indicates if this request hit the SSL session ticket and avoided a full SSL handshake. cqssv The SSL version used to communicate with the client. cqssc The cipher used by Trafﬁc Server to communicate with the client over SSL. cqtx The full HTTP client request text, minus headers; for example,
GET http://www.company.com HTTP/1.0
In reverse proxy mode, Trafﬁc Server logs the rewritten/mapped URL (according to the rules in remap. config), _not_ the pristine/unmapped URL. cqu The universal resource identiﬁer (URI) of the request from client to Trafﬁc Server (subset of cqtx ). In reverse proxy mode, Trafﬁc Server logs the rewritten/mapped URL (according to the rules in remap. config), _not_ the pristine/unmapped URL. cquc The client request canonical URL. This differs from cqu in that blanks (and other characters that might not be parsed by log analysis tools) are replaced by escape sequences. The escape sequence is a percentage sign followed by the ASCII code number in hex. See cquuc. cqup The client request URL path; speciﬁes the argument portion of the URL (everything after the host). For example, if the URL is http://www.company.com/images/x.gif, then this ﬁeld displays /images/ x.gif See cquup. cqus The client request URL scheme. cquuc The client request unmapped URL canonical. This ﬁeld records a URL before it is remapped (reverse proxy mode). cquup The client request unmapped URL path. This ﬁeld records a URL path before it is remapped (reverse proxy mode).
188 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest cquuh The client request unmapped URL host. This ﬁeld records a URL’s host before it is remapped (reverse proxy mode). cluc The cache lookup URL, or cache key, for the client request. This URL is canonicalized as well. crat The Retry-After time in seconds, if speciﬁed by the origin server. crc The cache result code; speciﬁes how the cache responded to the request (HIT, MISS, and so on). chm The cache hit-miss status, specifying which level of the cache this was served out of. This is useful for example to show whether it was a RAM cache vs disk cache hit. Future versions of the cache will support more levels, but right now it only supports RAM (HIT_RAM) vs rotational disk (HIT_DISK). csscl The cached response length (in bytes) from origin server to Trafﬁc Server. csshl The cached header length in the origin server response to Trafﬁc Server (in bytes). csshv The cached server response HTTP version (1.0, 1.1, etc.). csssc The cached HTTP response status code from origin server to Trafﬁc Server. cwr The cache write result (-, WL_MISS, INTR`, ERR or FIN) cwtr The cache write transform result fsiz The size of the ﬁle (n bytes) as seen by the origin server. {Milestone field name}ms The timestamp in milliseconds of a speciﬁc milestone for this request. see TSHttpTxnMilestoneGet() for milestone names. {Milestone field name1-Milestone field name2}msdms The difference in milliseconds of be- tween two milestones. see TSHttpTxnMilestoneGet() for milestone names. pfsc The proxy ﬁnish status code; speciﬁes whether the Trafﬁc Server request to the origin server was successfully completed (FIN), interrupted (INTR) or timed out (TIMEOUT). phn The hostname of the Trafﬁc Server that generated the log entry in collated log ﬁles. phi The IP of the Trafﬁc Server that generated the log entry in collated log ﬁles. phr The proxy hierarchy route; the route Trafﬁc Server used to retrieve the object. php The TCP port number that Trafﬁc Server served this request from. piid The plugin ID for the transaction. This is set for plugin driven transactions via TSHttpConnectWithPluginId(). pitag The plugin tag for the transaction. This is set for plugin driven transactions via TSHttpConnectWithPluginId(). pqbl The proxy request transfer length; the body length in Trafﬁc Server’s request to the origin server. pqhl The proxy request header length; the header length in Trafﬁc Server’s request to the origin server. pqsi The proxy request server IP address (0 on cache hits and parent-ip for requests to parent proxies). pqsn The proxy request server name; the name of the server that fulﬁlled the request. pscl The length of the Trafﬁc Server response to the client (in bytes). psct The content type of the document from server response header: (for example, img/gif ). pshl The header length in Trafﬁc Server‘s response to the client. psql The proxy response transfer length in Squid format (includes header and content length). pssc The HTTP response status code from Trafﬁc Server to the client. pqssl Indicates whether the connection from Trafﬁc Server to the origin was over SSL or not.
3.8. Event and Error Monitoring 189 Apache-Trafﬁcserver-Server Documentation, latest sca The number of attempts in the transaction Trafﬁc Server tries to connect to the origin server. shi The IP address resolved from the DNS name lookup of the host in the request. For hosts with multiple IP addresses, this ﬁeld records the IP address resolved from that particular DNS lookup. This can be misleading for cached documents. For example: if the ﬁrst request was a cache miss and came from IP1 for server S and the second request for server S resolved to IP2 but came from the cache, then the log entry for the second request will show IP2. shn The hostname of the origin server. sscl The response length (in bytes) from origin server to Trafﬁc Server. sshl The header length (in bytes) in the origin server response to Trafﬁc Server. sshv The server response HTTP version (1.0, 1.1, etc.). sssc The HTTP response status code from origin server to Trafﬁc Server. stms The time spent accessing the origin (in milliseconds); the time is measured from the time the connection with the origin is established to the time the connection is closed. stmsh Same as stms but in hexadecimal. stmsf The time Trafﬁc Server spends accessing the origin as a fractional number of seconds. That is, the time is formated as a ﬂoating-point number, instead of an integer as in stms. For example: if the time is 1500 milliseconds, then this ﬁeld displays 1.5 while the stms ﬁeld displays 1500 and the sts ﬁeld displays 1. sts The time Trafﬁc Server spends accessing the origin, in seconds. sstc The number of transactions between Trafﬁc Server and the origin server from a single server session. A value greater than 0 indicates connection reuse. ttms The time Trafﬁc Server spends processing the client request; the number of milliseconds between the time the client establishes the connection with Trafﬁc Server and the time Trafﬁc Server sends the last byte of the response back to the client. ttmsh Same as ttms but in hexadecimal. ttmsf The time Trafﬁc Server spends processing the client request as a fractional number of seconds. Time is speciﬁed in millisecond resolution; however, instead of formatting the output as an integer (as with ttms), the display is formatted as a ﬂoating-point number representing a fractional number of seconds. For example: if the time is 1500 milliseconds, then this ﬁeld displays 1.5 while the ttms ﬁeld displays 1500 and the tts ﬁeld displays 1. tts The time Trafﬁc Server spends processing the client request; the number of seconds between the time at which the client establishes the connection with Trafﬁc Server and the time at which Trafﬁc Server sends the last byte of the response back to the client.
Custom Field Cross-Reference
The following sections illustrate the correspondence between Trafﬁc Server custom logging ﬁelds and standard logging ﬁelds for the Squid and Netscape formats.
Squid
The following is a list of the Squid logging ﬁelds and the corresponding logging ﬁeld symbols.
190 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Squid Field Symbols time cqts elapsed ttms client chi action/code crc/pssc size psql method cqhm url cquc ident caun hierarchy/from phr/pqsn content psct This is the equivalent XML conﬁguration for the log above:
  %%%/%%%% %%/%%"/> 
Netscape Common
The following is a list of the Netscape Common logging ﬁelds and the corresponding Trafﬁc Server logging ﬁeld symbols. Netscape Common Field Symbols host chi usr caun [time] [cqtn] “req” “cqtx“ s1 pssc c1 pscl This is the equivalent XML conﬁguration for the log above:
   -% [%] \"%\" %%"/> 
Netscape Extended
The following table lists the Netscape Extended logging ﬁelds and the corresponding Trafﬁc Server logging ﬁeld symbols.
3.8. Event and Error Monitoring 191 Apache-Trafﬁcserver-Server Documentation, latest
Netscape Extended Field Symbols host chi usr caun [time] [cqtn] “req” “cqtx“ s1 pssc c1 pscl s2 sssc c2 sscl b1 cqbl b2 pqbl h1 cqhl h2 pshl h3 pqhl h4 sshl xt tts This is the equivalent XML conﬁguration for the log above:
   -% [%] \"%\" %% %%%%%%%%%"/> 
Netscape Extended-2
The following is a list of the Netscape Extended-2 logging ﬁelds and the corresponding Trafﬁc Server logging ﬁeld symbols. Netscape Extended-2 Field Symbols host chi usr caun [time] [cqtn] "req" "cqtx" s1 pssc c1 pscl s2 sssc c2 sscl b1 cqbl b2 pqbl h1 cqhl h2 pshl h3 pqhl h4 sshl xt tts route phr pfs cfsc ss pfsc crc crc This is the equivalent XML conﬁguration for the log above:
192 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
   -% [%] \"%\" %% %%%%%%%%%%
˓→%%%"/> 
Log Field Slicing
It is sometimes desirable to slice a log ﬁeld to limit the length of a given log ﬁeld’s output. Log Field slicing can be speciﬁed as below:
% %<{field}container[start:end]>
Omitting the slice notation defaults to the entire log ﬁeld. Slice notation only applies to a log ﬁeld that is of type string and can not be applied to IPs or timestamp which are converted to string from integer. The below slice speciﬁers are allowed. [start:end] Log ﬁeld value from start through end-1 [start:] Log ﬁeld value from start through the rest of the string [:end] Log ﬁeld value from the beginning through end-1 [:] Default - entire Log ﬁeld Some examples below
'%'// the whole characters of. '%[:]'// the whole characters of. '%'// the first 30 characters of. '%'// the last 10 characters of. '%'// everything except the last5 characters of.
Cache Result Codes
The following table describes the cache result codes in Squid and Netscape log ﬁles. TCP_HIT A valid copy of the requested object was in the cache and Trafﬁc Server sent the object to the client. TCP_MISS The requested object was not in cache, so Trafﬁc Server retrieved the object from the origin server (or a parent proxy) and sent it to the client. TCP_REFRESH_HIT The object was in the cache, but it was stale. Trafﬁc Server made an if-modified-since request to the origin server and the origin server sent a 304 not-modiﬁed response. Trafﬁc Server sent the cached object to the client. TCP_REF_FAIL_HIT The object was in the cache but was stale. Trafﬁc Server made an if-modified-since request to the origin server but the server did not respond. Trafﬁc Server sent the cached object to the client. TCP_REFRESH_MISS The object was in the cache but was stale. Trafﬁc Server made an if-modified-since request to the origin server and the server returned a new object. Trafﬁc Server served the new object to the client.
3.8. Event and Error Monitoring 193 Apache-Trafﬁcserver-Server Documentation, latest
TCP_CLIENT_REFRESH The client issued a request with a no-cache header. Trafﬁc Server obtained the re- quested object from the origin server and sent a copy to the client. Trafﬁc Server deleted the previous copy of the object from cache. TCP_IMS_HIT The client issued an if-modified-since request and the object was in cache and fresher than the IMS date, or an if-modified-since request to the origin server revealed the cached object was fresh. Trafﬁc Server served the cached object to the client. TCP_IMS_MISS The client issued an if-modified-since request and the object was either not in cache or was stale in cache. Trafﬁc Server sent an if-modified-since request to the origin server and received the new object. Trafﬁc Server sent the updated object to the client. TCP_SWAPFAIL The object was in the cache but could not be accessed. The client did not receive the object. ERR_CLIENT_ABORT The client disconnected before the complete object was sent. ERR_CONNECT_FAIL Trafﬁc Server could not reach the origin server. ERR_DNS_FAIL The Domain Name Server (DNS) could not resolve the origin server name, or no DNS could be reached. ERR_INVALID_REQ The client HTTP request was invalid. (Trafﬁc Server forwards requests with unknown methods to the origin server.) ERR_READ_TIMEOUT The origin server did not respond to Trafﬁc Server‘s request within the timeout interval. ERR_PROXY_DENIED Client service was denied. ERR_UNKNOWN The client connected, but subsequently disconnected without sending a request.
Summary Logs
Due to the speed and efﬁciency of Trafﬁc Server, a heavily-loaded node will generate many events and the event logs can quickly grow to very large sizes. Using SQL-like aggregate operators, you can conﬁgure Trafﬁc Server to create summary log ﬁles that summarize a set of log entries over a speciﬁed period of time. This can signiﬁcantly reduce the size of the log ﬁles generated. To generate a summary log ﬁle, create a LogFormat object in the XML-based logging conﬁguration logs_xml. config using the SQL-like aggregate operators below. You can apply each of these operators to speciﬁc ﬁelds, over a speciﬁed interval. • COUNT • SUM • AVERAGE • FIRST • LAST To create a summary log ﬁle format: 1. Deﬁne the format of the log ﬁle in logs_xml.config as follows:
   :%"/>  
194 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Where operator is one of the ﬁve aggregate operators (COUNT, SUM, AVERAGE, FIRST, LAST); field is the logging ﬁeld you want to aggregate; and n is the interval (in seconds) between summary log entries. You can specify more than one operator in the format line. For more information, refer to logs_xml. config. 2. Run the command traffic_line -x to apply conﬁguration changes . The following example format generates one entry every 10 seconds. Each entry contains the timestamp of the last entry of the interval, a count of the number of entries seen within that 10-second interval, and the sum of all bytes sent to the client:
   :% :%"/>  
: You cannot create a format speciﬁcation that contains both aggregate operators and regular ﬁelds. For example, the following speciﬁcation would be invalid:
 :% :% :%"/>
Log Collation
You can use the Trafﬁc Server log ﬁle collation feature to collect all logged information in one place. Log collation enables you to analyze a set of Trafﬁc Server clustered nodes as a whole (rather than as individual nodes) and to use a large disk that might only be located on one of the nodes in the cluster. Trafﬁc Server collates log ﬁles by using one or more nodes as log collation servers and all remaining nodes as log collation clients. When a Trafﬁc Server node generates a buffer of event log entries, it ﬁrst determines if it is the collation server or a collation client. The collation server node writes all log buffers to its local disk, just as it would if log collation was not enabled. Log collation servers can be standalone or they can be part of a node running Trafﬁc Server. The collation client nodes prepare their log buffers for transfer across the network and send the buffers to the log collation server. When the log collation server receives a log buffer from a client, it writes it to its own log ﬁle as if it was generated locally. For a visual representation of this, see the ﬁgure below. If log clients cannot contact their log collation server, then they write their log buffers to their local disks, into orphan log ﬁles. Orphaned log ﬁles require manual collation.
: Log collation can have an impact on network performance. Because all nodes are forwarding their log data buffers to the single collation server, a bottleneck can occur. In addition, collated log ﬁles contain timestamp information for each entry, but entries in the ﬁles do not appear in strict chronological order. You may want to sort collated log ﬁles before doing analysis.
Server Conﬁguration
To conﬁgure a Trafﬁc Server node to be a collation server, perform the following conﬁguration adjustments in records.config:
3.8. Event and Error Monitoring 195 Apache-Trafﬁcserver-Server Documentation, latest
3.15: Log collation
1. Set :ts:cv:‘proxy.local.log.collation_mode‘ to 1 to indicate this node will be a server.
CONFIG proxy.local.log.collation_mode INT1
2. Conﬁgure the port on which the server will listen to incoming collation transfers from clients, using :ts:cv:‘proxy.conﬁg.log.collation_port‘. If omitted, this defaults to port 8085.
CONFIG proxy.config.log.collation_port INT 8085
3. Conﬁgure the shared secret used by collation clients to authenticate their sessions, using :ts:cv:‘proxy.conﬁg.log.collation_secret‘.
CONFIG proxy.config.log.collation_secret STRING"seekrit"
4. Run the command traffic_line -x to apply the conﬁguration changes.
: If you modify the collation_port or secret after connections between the collation server and collation clients have been established, then you must restart Trafﬁc Server on all nodes.
Standalone Collator
If you do not want the log collation server to be a Trafﬁc Server node, then you can install and conﬁgure a standalone collator (SAC) that will dedicate more of its power to collecting, processing, and writing log ﬁles. To install and conﬁgure a standalone collator: 1. Conﬁgure your Trafﬁc Server nodes as log collation clients based on the instructions in Client Conﬁguration.
196 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
2. Copy the traffic_sac binary from the Trafﬁc Server bin directory, and place in a suitable location on the system that will act as the standalone collator. 3. Copy the libtsutil.so libraries from the Trafﬁc Server lib directory to the machine serving as the stan- dalone collator. 4. Create a directory called config in the directory that contains the traffic_sac binary. 5. Create a directory called internal in the config directory you created above. This directory is used inter- nally by the standalone collator to store lock ﬁles. 6. Copy the records.config ﬁle from a Trafﬁc Server node conﬁgured to be a log collation client to the config directory you created on the standalone collator. The records.config ﬁle contains the log collation secret and the port you speciﬁed when conﬁguring Trafﬁc Server nodes to be collation clients. The collation port and secret must be the same for all collation clients and servers. 7. Edit :ts:cv:‘proxy.conﬁg.log.logﬁle_dir‘ in records.config to specify a location on your standalone col- lator where the collected log ﬁles should be stored.
CONFIG proxy.config.log.logfile_dir STRING"/var/log/trafficserver/"
8. Enter the following command to start the standalone collator process:
 traffic_sac-c config
You will likely want to conﬁgure this program to run at server startup, as well as conﬁgure a service monitor in the event the process terminates abnormally. Please consult your operating system’s documentation for how to achieve this.
Client Conﬁguration
To conﬁgure a Trafﬁc Server node to be a collation client, follow the steps below. If you modify the collation_port or secret after connections between the collation clients and the collation server have been established, then you must restart Trafﬁc Server. 1. In the records.config ﬁle, edit the following variables: • :ts:cv:‘proxy.local.log.collation_mode‘: 2 to conﬁgure this node as log collation client and sen standard formatted log entries to the collation server. For XML-based formatted log entries, see logs_xml. config ﬁle. • :ts:cv:‘proxy.conﬁg.log.collation_host‘ • :ts:cv:‘proxy.conﬁg.log.collation_port‘ • :ts:cv:‘proxy.conﬁg.log.collation_secret‘ • :ts:cv:‘proxy.conﬁg.log.collation_host_tagged‘ • :ts:cv:‘proxy.conﬁg.log.max_space_mb_for_orphan_logs‘ 2. Run the command traffic_line -x to apply the conﬁguration changes.
Collating Custom Logs
If you use custom event log ﬁles, then you must edit logs_xml.config, in addition to conﬁguring a collation server and collation clients.
3.8. Event and Error Monitoring 197 Apache-Trafﬁcserver-Server Documentation, latest
To collate custom event log ﬁles: 1. On each collation client, edit logs_xml.config and add the CollationHosts attribute to the LogObject speciﬁcation:
    
Where ipaddress is the hostname or IP address of the collation server to which all log entries (for this object) are forwarded, and port is the port number for communication between the collation server and collation clients. 2. Run the command traffic_line -L to restart Trafﬁc Server on the local node or traffic_line -M to restart Trafﬁc Server on all the nodes in a cluster.
ASCII Log Pipes
In addition to ASCII and BINARY ﬁle modes for custom log formats, Trafﬁc Server can output log entries in ASCII_PIPE mode. This mode writes the log entries to a UNIX named pipe (a buffer in memory). Other pro- cesses may read from this named pipe using standard I/O functions. The advantage of this mode is that Trafﬁc Server does not need to write the entries to disk, which frees disk space and bandwidth for other tasks. When the buffer is full, Trafﬁc Server drops log entries and issues an error message indicating how many entries were dropped. Because Trafﬁc Server only writes complete log entries to the pipe, only full records are dropped.
Event Log Builder
If you need any assistance building your event log, you can try out our online log builder. This is a work in progress, so any comments, critique or suggestions are most welcome.
Trafﬁc Manager Alarms
Trafﬁc Server signals an alarm when it detects a problem. For example, the space allocated to event logs could be full or Trafﬁc Server may not be able to write to a conﬁguration ﬁle.
Email Alarms
To conﬁgure Trafﬁc Server to send an email to a speciﬁc address whenever an alarm occurs, follow the steps below: 1. Set :ts:cv:‘proxy.conﬁg.alarm_email‘ in records.config to the email address you want to receive alarm notiﬁcations.
CONFIG proxy.config.alarm_email STRING"[email protected]"
2. Run the command traffic_ctl config reload to apply the conﬁguration changes.
198 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Using a Script File for Alarms
Alarm messages are built into Trafﬁc Server and cannot be changed. However, you can write a script ﬁle to execute cer- tain actions when an alarm is signaled. Trafﬁc Server provides a sample script ﬁle named example_alarm_bin. sh in the bin directory which can serve as the basis for your custom alarm scripts.
Viewing Statistics from Trafﬁc Line
You can use the Trafﬁc Line command-line interface to view statistics about Trafﬁc Server performance and web trafﬁc. In addition to viewing statistics, you can also conﬁgure, stop, and restart the Trafﬁc Server system. For additional information, refer to Conﬁgure Trafﬁc Server Using Trafﬁc Line and traffic_line. You can view speciﬁc information about a Trafﬁc Server node or cluster by specifying the variable that corresponds to the statistic you want to see. To view a statistic, enter the following command::
 traffic_ctl metric get VARIABLE
 where variable is the variable representing the information you want to view. For a list of variables you can specify, refer to Trafﬁc Server Metrics. For example, the following command displays the document hit rate for the Trafﬁc Server node::
 traffic_ctl metric get proxy.node.cache_hit_ratio
If the Trafﬁc Server bin directory is not in your path, then prepend the Trafﬁc Line command with ./ (for example: traffic_ctl metric get VARIABLE).
Statistics
Statistics provide a live view into the current state of Trafﬁc Server and many of the plugins which may be usd with it. They can, and generally should be, used as a core piece in any performance and operational monitoring infrastructure as they expose a much greater detail about the workings of Trafﬁc Server than can reliably be obtained through any other means.
Accessing Statistics
There are currently two methods provided with Trafﬁc Server to view statistics: Trafﬁc Line and Stats Over HTTP.
Trafﬁc Line
The command line utility traffic_line offers a text based interface for viewing Trafﬁc Server statistics. Invoca- tion is simple and requires knowing the speciﬁc name of the statistic you wish you view:
 traffic_line-r
You may notice that this is the same utility, and argument, used for viewing conﬁguration variables of a running Trafﬁc Server instance. Unlike conﬁguration variables, you cannot modify a statistic value with the traffic_line program. This utility is enabled and built by default, and will be located in the bin/ subdirectory of your Trafﬁc Server installation. There are no required changes to your conﬁguration to allow traffic_line to function, however it may only be run by users with permissions to access the Trafﬁc Server Unix socket. This will typically limit use to
3.8. Event and Error Monitoring 199 Apache-Trafﬁcserver-Server Documentation, latest
 root as well as the system user you have conﬁgured Trafﬁc Server to run under or any other system users which share the same group as you have conﬁgured Trafﬁc Server to use.
Stats Over HTTP
Trafﬁc Server includes a stable plugin, Stats Over HTTP Plugin, which provides HTTP access to all Trafﬁc Server statistcs. The plugin returns a JSON object with all statistics and their current values. It is not possible to return a subset of the statistics. The plugin must be enabled before you may use it.
Enabling Stats Over HTTP
To enable the Stats Over HTTP Plugin plugin, you must add the following to your plugin.config:
 stats_over_http.so
Once the plugin is enabled and Trafﬁc Server has reloaded, you can test that it is working properly by issuing a simple HTTP request with curl. Assuming your Trafﬁc Server installation is using the default interface and port bindings, running the following command on the same host as Trafﬁc Server should now work:
 curl http://localhost:8080/_stats
You should be presented with an HTTP response containing a single JSON object which lists all the available statistics and their current values. If you have conﬁgured Trafﬁc Server to only listen on a speciﬁc interface, or to use a different port, you may need to adjust the URL in the command above. If you wish to have the stats made available at a non-default path, then that path should be given as the sole argument to the plugin, as so:
 stats_over_http.so 81c075bc0cca1435ea899ba4ad72766b
The above plugin.config entry will result in your Trafﬁc Server statistics being located at / 81c075bc0cca1435ea899ba4ad72766b on any host and port on which you have your Trafﬁc Server instance listening.
Statistics Security and Privacy
Simply changing the path at which your statistics are available should be considered very weak security. While cache objects themselves cannot be accessed through the plugin’s JSON output, and no modiﬁcations to the conﬁguration or operation of Trafﬁc Server may be made through the plugin, the statistics may reveal much more about your network’s trafﬁc and architecture than you wish to be publicly available. A better method is to use an ACL Filter in remap.config to restrict access to clients. For instance, if your Trafﬁc Server host resides on a private network in the 10.1.1.0/24 IPv4 address space listening on the address 10.1.1.10, separate from its public interface(s) used to serve client requests, you could add the following remap conﬁguration:
 map http://10.1.1.10/_stats http://10.1.1.10/_stats @action=deny @src_ip=0.0.0.0-255.
˓→255.255.255 map http://10.1.1.10/_stats http://10.1.1.10/_stats @action=allow @src_ip=10.1.1.0-10.
˓→1.1.255
The above conﬁguration sets the default policy for the entirety of IPv4 address space to deny, but then exempts the 10.1.1.0/24 network by permitting their requests to be processed by Trafﬁc Server. If your monitoring infrastructure makes use of locally-installed data collection agents, you may even wish to restrict access to the Stats Over HTTP plugin to all but localhost.
200 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Core Statistics
Core statistics are available regardless of which plugins are enabled. This section breaks them down into categories based on the general Trafﬁc Server function or component to which they’re related.
General
Logging
Cache
Cache Volume
Each conﬁgured cache volume receives a complete set of statistics from Trafﬁc Server. They are differentiated from each other by the incrementing number that is part of the statistic name. The volume number begins at 0 and increments by 1 for each additional volume. The statistics are documented in this section using the default volume number in a conﬁguration with only one cache volume: 0.
Hierarchical Cache
HostDB
The HostDB component of Trafﬁc Server manages origin server name resolution and host health checks. The statistics documented here can help to ensure that your Trafﬁc Server instances are not spending an unreasonable amount of timing resolving your origin servers’ hostnames prior to object revalidation or retrieval.
DNS
Cluster
The statistics documented in this section relate to Trafﬁc Server clusters, which are synchronized sets of Trafﬁc Server instances running across multiple hosts. For more information on conﬁguring and using the clustering feature, please refer to the Trafﬁc Server Cluster chapter of the Administrator’s Guide.
Origin Server
Network I/O
SSL/TLS
SSL Cipher
The following statistics provide individual counters for the number of client requests which were satisﬁed using the given SSL cipher.
3.8. Event and Error Monitoring 201 Apache-Trafﬁcserver-Server Documentation, latest
HTTP Transaction
HTTP Response Code
The statistics documented in this section provide counters for the number of HTTP responses delivered to user agents by your Trafﬁc Server instance, grouped by the HTTP status code of the response. Please refer to the HTTP Status Codes appendix for more details on what each status code means.
HTTP Request Method
The statistics documented in this section provide counters for the number of incoming requests to the Trafﬁc Server instance, broken down by the HTTP method of the request.
HTTP Connection
HTTP Document Size
HTTP Header
Bandwidth and Transfer
Congestion
SOCKS
WebSocket
Miscellaneous
Plugin Speciﬁc Statistics
Monitoring
Using Built-in Tools
Integrating Third-Party Tools
There are many monitoring and alerting systems available, too many for us to hope to cover every possible option here. We can, however, attempt to document using some of the more common options. Some of these third party monitoring applications and services are able to tap into the extensive list of Trafﬁc Server statistics, others make use of Trafﬁc Server log ﬁles, and yet others are aimed at simple health check reporting. An ex- tensive service monitoring conﬁguration may take advantage of more than one service to play to their complementary strengths. Which tool, or combination of tools, is right for your infrastructure will likely vary.
202 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Circonus
Circonus is a commercial systems monitoring and analytics product, available in hosted (SaaS) and on-site conﬁgura- tions. While it does not currently offer a module speciﬁc to Apache Trafﬁc Server™ statistics, it is very easy to add moni- toring by combining the Trafﬁc Server plugin Stats Over HTTP with Circonus’s JSON metrics collector.
Setup
The following steps assume that you already have an active Circonus account on their hosted service, or that you already have an on-site installation. If you do not, that must be set up ﬁrst. Trial accounts are available and will sufﬁce for this guide.
Enable Stats Over HTTP
Follow the directions in the Stats Over HTTP Plugin chapter to conﬁgure the plugin. At a bare minimum, you will need to add the plugin to plugin.config. For this guide, we will assume the following entry: stats_over_http.so _statistics
This enables the plugin and makes Trafﬁc Server statistics available at the path /_statistics. For production environments, you are strongly encouraged to use a different (and obfuscated) path for your statistics since they reveal many internal details of your caching proxy. The plugin documentation provides more details.
Create Circonus Check
For these steps, we will assume that Trafﬁc Server is listening on port 80 at the domain trafficserver. example.com. You should adjust these to match your environment. 1. Begin the new check creation process from within your Circonus account by clicking the New Check button near the top-right of the checks screen.
2. For the check type, you should select JSON under the Custom list, and then choose the Pull type. The broker you choose to use is entirely up to you and will depend largely on whether you are using on-site Circonus or the hosted service, as well as the geographic location of your Trafﬁc Server instance(s).
3.8. Event and Error Monitoring 203 Apache-Trafﬁcserver-Server Documentation, latest
3. Click on Conﬁgure Check to proceed. 4. Click on the Show Advanced Conﬁguration link to expand the check options. Enter your Trafﬁc Server instance’s externally-accessible domain for Host and make sure to set URI to the path at which the statistics are available. Adjust any of the other options to match your environment if necessary (for this guide, only Host and URI will need to be entered).
204 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
5. Click on Test Check to proceed. 6. If all is well, you should see a long list (several hundred entries) of Trafﬁc Server statistics, with their associated values. If you do not, verify that Trafﬁc Server is running and use curl to manually fetch the statistics data from your server.
3.8. Event and Error Monitoring 205 Apache-Trafﬁcserver-Server Documentation, latest
7. You may want to limit the metrics you actually track from Trafﬁc Server, since so many are made available. If so, simply uncheck those you aren’t interested in from the list. If you only want to track a few metrics, click Select None and then check the ones you want. You can ﬁlter the results to easily ﬁnd metrics in the list. 8. Save your new check. 9. Verify the collected metrics in Circonus by opening your new check and clicking on Metrics Grid for an overview visualization of all the data being collected.
Congratulations! You’re now ready to begin setting up alerts, visualizations, and dashboards for your Trafﬁc Server statistics. You can repeat the above steps for any additional Trafﬁc Server instances you have running.
Further Reading
• Ofﬁcial Circonus Website
206 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Logstash
Logstash is a powerful, open source, unstructured data processing program that can accept text data from many differ- ent sources (directly over TCP/UDP, via Unix sockets, or by reading in ﬁles from disk for example), in many different formats and transform those inputs into structured, searchable documents. One of the most common use cases is to take text based system and application logs, extract individual ﬁelds (e.g. host names, error codes, timing data, and so on), and make the data available in Elasticsearch for searching and reporting. In this guide, we will cover the basics of getting your Trafﬁc Server log data into Logstash. Going the next step and building fancy Kibana dashboards on top of that is currently left as an exercise for the reader.
Trafﬁc Server Log Formats
Trafﬁc Server provides a very ﬂexible set of logging outputs. Almost any format can be constructed. The full range of options is covered in the Logging chapter. This guide will walk you through using the appropriate ﬁlters in Logstash for the common logging formats in Trafﬁc Server. If you have constructed your own custom log formats, you will need to build upon these examples and refer to the Logstash documentation to produce custom ﬁlters capable of parsing your own formats.
Logstash Input
For the on-disk logs produced by Trafﬁc Server, you will want to use Logstash’s file input plugin. Note that your logs must be in ASCII format, not binary, for the plugin to work. Assuming that your Trafﬁc Server event logs are named access-.log and stored at /var/log/trafficserver/, the following Logstash input conﬁguration should work: input{ file { path=>/var/log/trafficserver/access- *.log } }
Logstash provides some additional tweaking options, which are explained in the ﬁle plugin documentation but the above provides the bare minimum required to have Logstash read log data from local disks.
Logstash Filters
The grok ﬁlter in Logstash allows you to completely tailor the parsing of your source data and extract as many or as few ﬁelds as you like. Some patterns are already built and can be used very easily. If you have built custom log formats for Trafﬁc Server, you may need to write your own patterns, however.
Squid Compatible
The Squid log format includes, unsurprisingly, a few useful ﬁelds for proxy servers. Using the following grok pattern will extract this information from your Trafﬁc Server logs if you employ the Squid compatible log format:
3.8. Event and Error Monitoring 207 Apache-Trafﬁcserver-Server Documentation, latest
 filter{ grok { match=>{"message"=>"%{NUMBER:timestamp}%{NUMBER:timetoserve}%
˓→{IPORHOST:clientip}%{WORD:cachecode}/%{NUMBER:response}%{NUMBER:bytes}%
˓→{WORD:verb}%{NOTSPACE:request}%{USER:auth}%{NOTSPACE:route}%{DATA:contenttype}"
˓→} } date { match=>["timestamp","UNIX"] } }
The resulting structured document will contain the following ﬁelds: Field Description times- Date and time of the client request. tamp time- Time, in seconds, from initial client connection to Trafﬁc Server until the last byte has been sent back toserve to client from Trafﬁc Server. clientip Client IP address or hostname. cachecode Cache Result Codes. response HTTP response status code sent by Trafﬁc Server to the client. bytes Length, in bytes, of the Trafﬁc Server response to the client, including headers. verb HTTP method (e.g. GET, POST, etc.) of the client request. request URL speciﬁed by the client request. auth Authentication username supplied by the client, if present. route Proxy hierarchy route; the route used by Trafﬁc Server to retrieve the cache object. content- Content type of the response. type
Netscape Common
If your Trafﬁc Server instance is already outputting Netscape Common format logs, then Logstash’s COMMONAPACHELOG pattern will handle your logs out of the box. Add the following ﬁlter block to your Logstash conﬁguration:
 filter{ grok { match=>{"message"=>"% {COMMONAPACHELOG}"} } }
This will produce a structured document for each log entry with the following ﬁelds:
208 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Field Description clientip Client IP address or hostname. ident Always a literal - character for Trafﬁc Server logs. auth The authentication username for the client request. A - means no authentication was required (or supplied). timestamp The date and time of the client request. verb HTTP method used for the request (e.g. GET, POST, etc.). request URL speciﬁed by the client request. httpver- HTTP version (e.g. 1.1) used by the client. sion rawre- See note below. quest response HTTP status code used for Trafﬁc Server response (not the origin’s response code). bytes Length of Trafﬁc Server response to client, in bytes.
: rawrequest is populated when the usual "  http/" pattern was not matched. In that event, those three ﬁelds will be missing from the document, and instead rawrequest will have the original string.
Netscape Extended
The following pattern adds to Common Apache to support the additional ﬁelds found in Netscape Extended: filter{ grok { match=>{"message"=>"% {COMMONAPACHELOG} %{NUMBER:originstatus}%
˓→{NUMBER:originrespbytes}%{NUMBER:clientreqbytes}%{NUMBER:proxyreqbytes}%
˓→{NUMBER:clienthdrbytes}%{NUMBER:proxyresphdrbytes}%{NUMBER:proxyreqhdrbytes}%
˓→{NUMBER:originhdrbytes}%{NUMBER:timetoserve}"} } }
Because this starts out with the COMMONAPACHELOG pattern, you will get all of the ﬁelds mentioned in Netscape Common above, as well as the following: Field Description originstatus HTTP status code returned by origin server. originresp- Body length, in bytes, of origin’s response to Trafﬁc Server. bytes clientre- Body length, in bytes, of client request to Trafﬁc Server. qbytes proxyre- Body length, in bytes, of Trafﬁc Server request to origin. qbytes clienthdr- Header length, in bytes, of client request to Trafﬁc Server. bytes proxyresph- Header length, in bytes, of Trafﬁc Server response to client. drbytes proxyreqhdr- Header length, in bytes, of Trafﬁc Server request to origin. bytes originhdr- Header length, in bytes, of origin’s response to Trafﬁc Server. bytes timetoserve Time, in seconds, from initial client connection to Trafﬁc Server until the last byte has been sent back to client from Trafﬁc Server.
3.8. Event and Error Monitoring 209 Apache-Trafﬁcserver-Server Documentation, latest
Further Reading
• Logstash Documentation • Grok Patterns • Elasticsearch Documentation
Error Messages
The following table lists messages that can appear in system log ﬁles. This list is not exhaustive; it simply describes common warning messages that can occur and which might require your attention.
Fatal Process Messages
Accept port is not between 1 and 65535. Please check configuration The port speci- ﬁed in records.config that accepts incoming HTTP requests is not valid. Self loop is detected in parent proxy configuration The name and port of the parent proxy match that of Trafﬁc Server. This creates a loop when Trafﬁc Server attempts to send the request to the parent proxy.
Process Warnings
 error: error_number Generic logging error. Bad cluster major version range  for node  connect failed Incompatible software versions causing a problem. Connect by disallowed client , closing The speciﬁed client is not allowed to con- nect to Trafﬁc Server; the client IP address is not listed in the ip_allow.config ﬁle. Could not rename log  to  System error when renaming log ﬁle during roll. Did  of backup; still to do  Congestion is approaching. Different clustering minor versions  for node  continuing Incompatible software versions are causing a problem. Log format symbol  not found Custom log format references a ﬁeld symbol that does not exist. Refer to Log Formats. Missing field for field marker Error reading a log buffer. Unable to open log file , errno= Cannot open the log ﬁle. Error accessing disk  Trafﬁc Server might have a cache read problem. You might need to replace the disk. Too many errors accessing disk : declaring disk bad Trafﬁc Server is not using the cache disk because it encountered too many errors. The disk might be corrupt and might have to be replaced. No cache disks specified in storage.config file: cache disabled The Trafﬁc Server storage.config ﬁle does not list any cache disks; Trafﬁc Server is running in proxy-only mode. You must add the disks you want to use for the cache to storage.config.
210 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Alarm Messages
[Rollback::Rollback] Config file is read-only:  Go to the Trafﬁc Server config directory and check the indicated ﬁle permissions; change if necessary. [Rollback::Rollback] Unable to read or write config file  Go to the Traf- ﬁc Server config directory and make sure the indicated ﬁle exists. Check permissions and modify if necessary. [Traffic Manager] Configuration File Update Failed:  Go to the Traf- ﬁc Server config directory and check the indicated ﬁle permissions; change if necessary. [Traffic Manager] Mgmt <==>Proxy conn. closed An informational message to inform you that the traffic_server process is down. Access logging suspended - configured space allocation exhausted. The space allo- cated to the event log ﬁles is full; you must either increase the space or delete some log ﬁles so that access logging to continue. To prevent this error, consider rolling log ﬁles more frequently and enabling the autodelete feature. Access logging suspended - no more space on the logging partition. The entire parti- tion containing the event logs is full; you must delete or move some log ﬁles to enable access logging to continue. To prevent this error, consider rolling log ﬁles more frequently and enabling the autodelete feature. Created zero length place holder for config file  Go to the Trafﬁc Server config directory and check the indicated ﬁle. If it is indeed zero in length, then use a backup copy of the conﬁguration ﬁle. Traffic Server could not open logfile  Check permissions for the indicated ﬁle and the logging directory. Traffic Server failed to parse line  of the logging config file  Check your custom log conﬁguration ﬁle; there could be syntax errors. Refer to Custom Logging Fields for correct custom log format ﬁelds. vip_config binary is not setuid root, manager will be unable to enable virtual ip addresses The traffic_manager process is not able to set virtual IP addresses. You must setuid root for the vip_config ﬁle in the Trafﬁc Server bin directory.
HTML Messages Sent to Clients
Trafﬁc Server returns detailed error messages to client browsers when there are problems with the HTTP transactions requested by the browser. These Trafﬁc Server response messages correspond to standard HTTP response codes, but provide more information. A list of the more frequently encountered HTTP response codes is provided in HTTP Status Codes. The error messages can be customized. The actual response is generated from a template. These templates are stored in ﬁles which means the errors responses can be customized by modifying these ﬁles. The default directory for the template ﬁles is PREFIX/body_factory/default but this can be changed by the conﬁguration variable :ts:cv:‘proxy.conﬁg.body_factory.template_sets_dir‘. All ﬁles in this directory are added to a lookup table which is consulted when the error message is generated. The name used for lookup is by default that listed in the following table. It can be overridden by :ts:cv:‘proxy.conﬁg.body_factory.template_base‘ which, if set, is a string that is prepended to the search name along with an underscore. For example, if the default lookup name is cache#read_error then by default the response will be generated from the template in the ﬁle named cache#read_error. If the template base name were set to “apache” then the lookup would look for a ﬁle named apache_cache#read_error in the template table. This can be used to switch out error message sets or, because this variable is overridable, to select an error message set based on data in the transaction. The text for an error message is processed as if it were a custom logging format which enables customization by values present in the transaction for which the error occurred.
3.8. Event and Error Monitoring 211 Apache-Trafﬁcserver-Server Documentation, latest
The following table lists the hard-coded Trafﬁc Server HTTP messages, with corresponding HTTP response codes and customizable ﬁles. Access Denied 403 You are not allowed to access the document at location URL. access#denied Cache Read Error 500 Error reading from cache; please retry request. cache#read_error Connection Timed Out 504 Too much time has elapsed since the server has sent data. timeout#inactivity Content Length Required 400 Could not process this request because Content-Length was not speci- ﬁed. request#no_content_length Cycle Detected 400 Your request is prohibited because it would cause an HTTP proxy cycle. request#cycle_detected Forbidden 403  is not an allowed port for SSL connections (you have made a request for a secure SSL connection to a forbidden port number). access#ssl_forbidden Host Header Required 400 An attempt was made to transparently proxy your request, but this attempt failed because your browser did not send an HTTP Host header. Manually conﬁgure your browser to use http:/ /: as the HTTP proxy. Alternatively, end users can upgrade to a browser that supports the HTTP Host header ﬁeld. interception#no_host Host Header Required 400 Because your browser did not send a Host HTTP header ﬁeld, the virtual host being requested could not be determined. To access the website correctly, you must upgrade to a browser that supports the HTTP Host header ﬁeld. request#no_host HTTP Version Not Supported 505 The origin server  is using an unsupported version of the HTTP protocol. response#bad_version Invalid Content Length 400 Could not process this request because the speciﬁed Content-Length was invalid (less than 0).. request#invalid_content_length Invalid HTTP Request 400 Could not process this  HTTP method request for URL. request#syntax_error Invalid HTTP Response 502 The host  did not return the document URL correctly. response#bad_response Malformed Server Response 502 The host  did not return the document URL correctly. response#bad_response Malformed Server Response Status 502 The host  did not return the document URL correctly. response#bad_response Maximum Transaction Time exceeded 504 Too much time has elapsed while transmitting document URL. timeout#activity No Response Header From Server 502 The host  did not return the document URL cor- rectly. response#bad_response Not Cached 504 This document was not available in the cache, and you (the client) only accept cached copies. cache#not_in_cache Not Found on Accelerator 404 The request for URL on host  was not found. Check the location and try again. urlrouting#no_mapping NULL 502 The host  did not return the document URL correctly. response#bad_response Proxy Authentication Required 407 Please log in with username and password. access#proxy_auth_required
212 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Server Hangup 502 The server  closed the connection before the transaction was completed. connect#hangup Temporarily Moved 302 The document you requested, URL, has moved to a new location. The new location is . redirect#moved_temporarily Transcoding Not Available 406 Unable to provide the document URL in the format requested by your browser. transcoding#unsupported Tunnel Connection Failed 502 Could not connect to the server . connect#failed_connect Unknown Error 502 The host  did not return the document URL correctly. response#bad_response Unknown Host 500 Unable to locate the server named ; the server does not have a DNS entry. Perhaps there is a misspelling in the server name or the server no longer exists; double-check the name and try again. connect#dns_failed Unsupported URL Scheme 400 Cannot perform your request for the document URL because the protocol scheme is unknown. request#scheme_unsupported
Conﬁguring Trafﬁc Server
Trafﬁc Server provides several options for conﬁguring the system.
Conﬁgure Trafﬁc Server Using Trafﬁc Line
Trafﬁc Line enables you to quickly and easily change your Trafﬁc Server conﬁguration via command-line interface.
View Conﬁguration Options in Trafﬁc Line
To view a conﬁguration setting, enter the following command:
 traffic_ctl config get VARIABLE
 where var is the variable associated with the conﬁguration option. For a list of variables, refer to Conﬁguration Variables.
Change Conﬁguration Options in Trafﬁc Line
To change the value of a conﬁguration setting, enter the following command:
 traffic_config set VARIABLE VALUE
 where var is the variable associated with the conﬁguration option and value is the value you want to use. For a list of the variables, see Conﬁguration Variables.
Conﬁgure Trafﬁc Server Using Conﬁguration Files
As an alternative to using Trafﬁc Line, you can change Trafﬁc Server conﬁguration options by manually editing speciﬁc variables in records.config.
3.9. Conﬁguring Trafﬁc Server 213 Apache-Trafﬁcserver-Server Documentation, latest
Trafﬁc Server must reread the conﬁguration ﬁles for any changes to take effect. This is done with traffic_ctl config reload. Some conﬁguration changes require a full restart of Trafﬁc Server. The following is a sample portion of records.config:
3.16: Sample records.conﬁg ﬁle
In addition to records.config, Trafﬁc Server provides other conﬁguration ﬁles that are used to conﬁgure speciﬁc features. You can manually edit all conﬁguration ﬁles as described in Conﬁguration Files.
Performance Tuning
Apache Trafﬁc Server™ in its default conﬁguration should perform suitably for running the included regression test suite, but will need special attention to both its own conﬁguration and the environment in which it runs to perform optimally for production usage. There are numerous options and strategies for tuning the performance of Trafﬁc Server and we attempt to document as many of them as possible in the sections below. Because Trafﬁc Server offers enough ﬂexibility to be useful for many caching and proxying scenarios, which tuning strategies will be most effective for any given use case may differ, as well as the speciﬁc values for various conﬁguration options.
214 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Before You Start
One of the most important aspects of any attempt to optimize the performance of a Trafﬁc Server installation is the ability to measure that installation’s performance; both prior to and after any changes are made. To that end, it is strongly recommended that you establish some means to monitor and record a variety of performance metrics: request and response speed, latency, and throughput; memory and CPU utilization; and storage I/O operations. Attempts to tune a system without being able to compare the impact of changes made will at best result in haphazard, feel good results that may end up having no real world impact on your customers’ experiences, and at worst may even result in lower performance than before you started. Additionally, in the all too common situation of budget constraints, having proper measurements of existing performance will greatly ease the process of focusing on those individual components that, should they require hardware expenditures or larger investments of employee time, have the highest potential gains relative to their cost.
Building Trafﬁc Server
While the default compilation settings for Trafﬁc Server will produce a set of binaries capable of serving most caching and proxying needs, there are some build options worth considering in speciﬁc environments.
Hardware Tuning
As with any other server software, efﬁcient allocation of hardware resources will have a signiﬁcant impact on Trafﬁc Server performance.
CPU Selection
Apache Trafﬁc Server™ uses a hybrid event-driven engine and multi-threaded processing model for handling incoming requests. As such, it is highly scalable and makes efﬁcient use of modern, multicore processor architectures.
Memory Allocation
Though Trafﬁc Server stores cached content within an on-disk host database, the entire Cache Directory is always maintained in memory during server operation. Additionally, most operating systems will maintain disk caches within system memory. It is also possible, and commonly advisable, to maintain an in-memory cache of frequently accessed content. The memory footprint of the Trafﬁc Server process is largely ﬁxed at the time of server startup. Your Trafﬁc Server systems will need at least enough memory to satisfy basic operating system requirements, as well as capacity for the cache directory, and any memory cache you wish to use. The default settings allocate roughly 10 megabytes of RAM cache for every gigabyte of disk cache storage, though this setting can be adjusted manually in records.config using the setting :ts:cv:‘proxy.conﬁg.cache.ram_cache.size‘. Trafﬁc Server will, under the default conﬁguration, adjust this automatically if your system does not have enough physical memory to accomodate the aforementioned target. Aside from the cost of physical memory, and necessary supporting hardware to make use of large amounts of RAM, there is little downside to increasing the memory allocation of your cache servers. You will see, however, no beneﬁt from sizing your memory allocation larger than the sum of your content (and index overhead).
Disk Storage
Except in cases where your entire cache may ﬁt into system memory, your cache nodes will eventually need to interact with their disks. While a more detailed discussion of storage stratiﬁcation is covered in Cache Partitioning below, very
3.10. Performance Tuning 215 Apache-Trafﬁcserver-Server Documentation, latest brieﬂy you may be able to realize gains in performance by separating more frequently accessed content onto faster disks (PCIe SSDs, for instance) while maintaining the bulk of your on-disk cache objects, which may not receive the same high volume of requests, on lower-cost mechanical drives.
Operating System Tuning
Apache Trafﬁc Server™ is supported on a variety of operating systems, and as a result the tuning strategies available at the OS level will vary depending upon your chosen platform.
General Recommendations
TCP Keep Alive
TCP Congestion Control Settings
Ephemeral and Reserved Ports
Jumbo Frames
Linux
FreeBSD
OmniOS / illumos
Mac OS X
Trafﬁc Server Tuning
Trafﬁc Server itself, of course, has many options you may want to consider adjusting to achieve optimal performance in your environment. Many of these settings are recorded in records.config and may be adjusted with the traffic_ctl config set command line utility while the server is operating.
CPU and Thread Optimization
Thread Scaling
By default, Trafﬁc Server creates 1.5 threads per CPU core on the host system. This may be adjusted with the following settings in records.config: • :ts:cv:‘proxy.conﬁg.exec_thread.autoconﬁg‘ • :ts:cv:‘proxy.conﬁg.exec_thread.autoconﬁg.scale‘ • :ts:cv:‘proxy.conﬁg.exec_thread.limit‘
216 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Thread Afﬁnity
On multi-socket servers, such as Intel architectures with NUMA, you can adjust the thread afﬁnity conﬁguration to take advantage of cache pipelines and faster memory access, as well as preventing possibly costly thread migrations across sockets. This is adjusted with :ts:cv:‘proxy.conﬁg.exec_thread.afﬁnity‘ in records.config.
CONFIG proxy.config.exec_thread.affinity INT1
Thread Stack Size
:ts:cv:‘proxy.conﬁg.thread.default.stacksize‘
Polling Timeout
If you are experiencing unusually or unacceptably high CPU utilization during idle workloads, you may consider adjusting the polling timeout with :ts:cv:‘proxy.conﬁg.net.poll_timeout‘:
CONFIG proxy.config.net.poll_timeout INT 60
Memory Optimization
:ts:cv:‘proxy.conﬁg.thread.default.stacksize‘ :ts:cv:‘proxy.conﬁg.cache.ram_cache.size‘
Disk Storage Optimization
:ts:cv:‘proxy.conﬁg.cache.force_sector_size‘ :ts:cv:‘proxy.conﬁg.cache.max_doc_size‘ :ts:cv:‘proxy.conﬁg.cache.target_fragment_size‘
Cache Partitioning
Network Tuning
:ts:cv:‘proxy.conﬁg.net.connections_throttle‘
Error responses from origins are conistent and costly
If error responses are costly for your origin server to generate, you may elect to have Trafﬁc Server cache these responses for a period of time. The default behavior is to consider all of these responses to be uncacheable, which will lead to every client request to result in an origin request. This behavior is controlled by both enabling the feature via :ts:cv:‘proxy.conﬁg.http.negative_caching_enabled‘ and setting the cache time (in seconds) with :ts:cv:‘proxy.conﬁg.http.negative_caching_lifetime‘.
CONFIG proxy.config.http.negative_caching_enabled INT1 CONFIG proxy.config.http.negative_caching_lifetime INT 10
3.10. Performance Tuning 217 Apache-Trafﬁcserver-Server Documentation, latest
SSL-Speciﬁc Options
:ts:cv:‘proxy.conﬁg.ssl.max_record_size‘ :ts:cv:‘proxy.conﬁg.ssl.session_cache‘ :ts:cv:‘proxy.conﬁg.ssl.session_cache.size‘
Thread Types
Logging Conﬁguration
Plugin Tuning
Common Scenarios and Pitfalls
While environments vary widely and Trafﬁc Server is useful in a great number of different situations, there are at least some recurring elements that may be used as shortcuts to identifying problem areas, or realizing easier performance gains.
Conﬁguration Files
 cache.conﬁg
 cache.config ( /usr/local/etc/trafficserver/ ) Trafﬁc Server web . : • Not to cache objects from speciﬁc IP addresses • How long to pin particular objects in the cache • How long to consider cached objects as fresh • Whether to ignore no-cache directives from the server
: After you modify the cache.config ﬁle, navigate to the Trafﬁc Server bin directory; then run the traffic_ctl config reload command to apply changes. When you apply the changes to a node in a cluster, Trafﬁc Server automatically applies the changes to all other nodes in the cluster.
Format
Each line in the cache.config ﬁle contains a caching rule. Trafﬁc Server recognizes three space-delimited tags:
 primary_destination=value secondary_specifier=value action=value
You can use more than one secondary speciﬁer in a rule. However, you cannot repeat a secondary speciﬁer. The following list shows the possible primary destinations with allowed values. dest_domain A requested domain name. Trafﬁc Server matches the domain name of the destination from the URL in the request. dest_host A requested hostname. Trafﬁc Server matches the hostname of the destination from the URL in the request. dest_ip A requested IP address. Trafﬁc Server matches the IP address of the destination in the request. url_regex A regular expression (regex) to be found in a URL.
218 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
The secondary speciﬁers are optional in the cache.config ﬁle. The following list shows possible secondary speciﬁers with allowed values. port A requested URL port. scheme A request URL protocol: http or https. prefix A preﬁx in the path part of a URL. suffix A ﬁle sufﬁx in the URL. method A request URL method: get, put, post, trace. time A time range, such as 08:00-14:00. src_ip A client IP address. internal A boolean value, true or false, specifying if the rule should match (or not match) a transaction originating from an internal API. This is useful to differentiate transaction originating from an ATS plugin. The following list shows possible actions and their allowed values. action One of the following values: • never-cache conﬁgures Trafﬁc Server to never cache speciﬁed objects. • ignore-no-cache conﬁgures Trafﬁc Server to ignore all Cache-Control: no-cache headers. • ignore-client-no-cache conﬁgures Trafﬁc Server to ignore Cache-Control: no-cache headers from client requests. • ignore-server-no-cache conﬁgures Trafﬁc Server to ignore Cache-Control: no-cache headers from origin server responses. • cluster-cache-local conﬁgures the cluster cache to allow for this content to be stored locally on every cluster node. cache-responses-to-cookies Change the style of caching with regard to cookies. This effectively overrides the conﬁguration parameter :ts:cv:‘proxy.conﬁg.http.cache.cache_responses_to_cookies‘ and uses the same values with the same semantics. The override happens only for requests that match. pin-in-cache Preserves objects in cache, preventing them from being overwritten. Does not affect objects that are determined not to be cacheable. This setting can have performance issues, and severely affect the cache. For instance, if the primary destination matches all objects, once the cache is full, no new objects could get written as nothing would be evicted. Similarly, for each cache-miss, each object would incur extra checks to determine if the object it would replace could be overwritten. The value is the amount of time you want to keep the object(s) in the cache. The following time formats are allowed: • d for days; for example: 2d • h for hours; for example: 10h • m for minutes; for example: 5m • s for seconds; for example: 20s • mixed units; for example: 1h15m20s revalidate For objects that are in cache, overrides the the amount of time the object(s) are to be considered fresh. Use the same time formats as pin-in-cache. ttl-in-cache Forces object(s) to become cached, as if they had a Cache-Control: max-age: header. Can be overruled by requests with cookies. The value is the amount of time object(s) are to be kept in the cache, regardless of Cache-Control response headers. Use the same time formats as pin-in-cache and revalidate.
3.11. Conﬁguration Files 219 Apache-Trafﬁcserver-Server Documentation, latest
Examples
The following example conﬁgures Trafﬁc Server to revalidate gif and jpeg objects in the domain mydomain.com every 6 hours, and all other objects in mydomain.com every hour. The rules are applied in the order listed. dest_domain=mydomain.com suffix=gif revalidate=6h dest_domain=mydomain.com suffix=jpeg revalidate=6h dest_domain=mydomain.com revalidate=1h
Force a speciﬁc regex to be in cache between 7-11pm of the server’s time for 26hours. url_regex=example.com/articles/popular.* time=19:00-23:00 ttl-in-cache=1d2h
Prevent objects from being evicted from cache: url_regex=example.com/game/.* pin-in-cache=1h congestion.conﬁg
The congestion.config ﬁle (by default, located in /usr/local/etc/trafficserver/) enables you to conﬁgure Trafﬁc Server to stop forwarding HTTP requests to origin servers when they become congested, and then send the client a message to retry the congested origin server later. After you modify the congestion. config ﬁle, navigate to the Trafﬁc Server bin directory; then run the traffic_ctl config reload com- mand to apply changes. When you apply the changes to a node in a cluster, Trafﬁc Server automatically applies the changes to all other nodes in the cluster. Trafﬁc Server uses the congestion.config ﬁle only if you enable the :ts:cv:‘proxy.conﬁg.http.congestion_control.enabled‘ option. You can create rules in the congestion.conﬁg ﬁle to specify: • Which origin servers Trafﬁc Server tracks for congestion. • The timeouts Trafﬁc Server uses, depending on whether a server is congested. • The page Trafﬁc Server sends to the client when a server becomes congested. • If Trafﬁc Server tracks the origin servers per IP address or per hostname.
Format
Each line in congestion.config must follow the format below. Trafﬁc Server applies the rules in the order listed, starting at the top of the ﬁle. Trafﬁc Server recognizes three space-delimited tags: primary_destination=value secondary_specifier=value action=value
The following list shows possible primary destinations with allowed values. dest_domain A requested domain name. dest_host A requested hostname. dest_ip A requested IP address. url_regex A regular expression (regex) to be found in a URL. The secondary speciﬁers are optional in the congestion.conﬁg ﬁle. The following list shows possible secondary spec- iﬁers with allowed values. You can use more than one secondary speciﬁer in a rule; however, you cannot repeat a secondary speciﬁer. port A requested URL port or range of ports.
220 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 prefix A preﬁx in the path part of a URL. The following list shows the possible tags and their allowed values. max_connection_failures Default: 5 The maximum number of connection failures allowed within the fail window described below before Trafﬁc Server marks the origin server as congested. fail_window Default: 120 seconds. The time period during which the maximum number of connection failures can occur before Trafﬁc Server marks the origin server as congested. proxy_retry_interval Default: 10 seconds. The number of seconds that Trafﬁc Server waits before contact- ing a congested origin server again. client_wait_interval Default: 300 seconds. The number of seconds that the client is advised to wait before retrying the congested origin server. wait_interval_alpha Default: 30 seconds The upper limit for a random number that is added to the wait interval. live_os_conn_timeout Default: 60 seconds. The connection timeout to the live (uncongested) origin server. If a client stops a request before the timeout occurs, then Trafﬁc Server does not record a connection failure. live_os_conn_retries Default: 2 The maximum number of retries allowed to the live (uncongested) origin server. dead_os_conn_timeout Default: 15 seconds. The connection timeout to the congested origin server. dead_os_conn_retries Default: 1 The maximum number of retries allowed to the congested origin server. max_connection Default: -1 The maximum number of connections allowed from Trafﬁc Server to the origin server. error_page Default: "congestion#retryAfter" The error page sent to the client when a server is con- gested. You must enclose the value in quotes; congestion_scheme Default: "per_ip" Speciﬁes if Trafﬁc Server applies the rule on a per-host ("per_host") or per-IP basis ("per_ip"). You must enclose the value in quotes. For example: if the server www.host1.com has two IP addresses and you use the tag value "per_ip", then each IP address has its own number of connection failures and is marked as congested independently. If you use the tag value "per_host" and the server www.host1.com is marked as congested, then both IP addresses are marked as congested.
Examples
The following congestion.config rule conﬁgures Trafﬁc Server to stop forwarding requests to the server www. host.com on port 80 (HTTP trafﬁc) if the server is congested, according to the timeouts speciﬁed. Trafﬁc Server uses the default tag values because no tag has been speciﬁed.
 dest_host=www.host.com port=80
You can use one or more tags in a rule, but each tag must have one value only. If you specify no tags in the rule, then Trafﬁc Server uses the default values. You can override any of the default tag values by adding conﬁguration variables at the end of records.config as follows:
CONFIG proxy.config.http.congestion_control.default.tag INT|STRING value
 where tag is one of the tags described in the list under congestion.config and value is the value you want to use.
3.11. Conﬁguration Files 221 Apache-Trafﬁcserver-Server Documentation, latest
For example:
CONFIG proxy.config.http.congestion_control.default.congestion_scheme STRING per_host
: Rules in the congestion.config ﬁle override the following variables in the records.config ﬁle: proxy.config.http.connect_attempts_max_retries proxy.config.http.connect_attempts_max_retries_dead_server proxy.config.http.connect_attempts_rr_retries proxy.config.http.connect_attempts_timeout proxy.config.http.down_server.cache_time proxy.config.http.down_server.abort_threshold hosting.conﬁg
The hosting.config ﬁle (by default, located in /usr/local/etc/trafficserver/) you to assign cache volumes to speciﬁc origin servers and/or domains so that you can manage cache space efﬁciently and restrict disk usage. For step-by-step instructions on partitioning the cache according to origin servers and/or domains, refer to Partitioning the Cache. Before you can assign cache volumes to speciﬁc origin servers and/or domains, you must ﬁrst partition your cache according to size and protocol in the volume.config ﬁle. After you modify hosting.conﬁg, navigate to the Trafﬁc Server bin directory and run traffic_ctl config reload to apply your changes. When you apply the changes to a node in a cluster, Trafﬁc Server automatically applies the changes to all other nodes in the cluster.
: The volume.config conﬁguration must be the same on all nodes in a cluster.
Format
Each line in the hosting.config ﬁle must have one of the following formats: hostname=HOST volume=NUMBERS domain=DOMAIN volume=NUMBERS where HOST is the fully-qualiﬁed hostname of the origin server whose content you want to store on a particular volume (for example, www.myhost.com); DOMAIN is the domain whose content you want to store on a particular partition(for example, mydomain.com); and NUMBERS is a comma-separated list of the partitions on which you want to store the content that belongs to the origin server or domain listed. The partition numbers must be valid numbers listed in the ﬁle:volume.conﬁg. Note: To allocate more than one partition to an origin server or domain, you must enter the partitions in a comma- separated list on one line, as shown in the example below. The hosting.config ﬁle cannot contain multiple entries for the same origin server or domain.
222 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Generic Partition
When conﬁguring the hosting.config ﬁle, you must assign a generic volume to use for content that does not belong to any of the origin servers or domains listed. If all volumes for a particular origin server become corrupt, Trafﬁc Server will also use the generic volume to store content for that origin server. The generic volume must have the following format:
 hostname=* volume=NUMBERS
 where NUMBERS is a comma-separated list of generic volumes.
Examples
The following example conﬁgures Trafﬁc Server to store content from the domain mydomain.com in volume 1 and content from www.myhost.com in volume 2. Trafﬁc Server stores content from all other origin servers in volumes 3 and 4.
 domain=mydomain.com volume=1 hostname=www.myhost.com volume=2 hostname=* volume=3,4
 icp.conﬁg
The icp.config ﬁle deﬁnes ICP peers (parent and sibling caches).
: After you modify the icp.conﬁg ﬁle, navigate to the Trafﬁc Server bin directory and run the traffic_ctl config reload command to apply the changes. When you apply the changes to a node in a cluster, Trafﬁc Server automatically applies the changes to all other nodes in the cluster.
Format
Each line in the icp.conﬁg ﬁle contains the name and conﬁguration information for a single ICP peer in the following format:
 host : host_IP : ctype : proxy_port : icp_port : MC_on : MC_IP : MC_TTL :
Each ﬁeld is described in the following list. host The hostname of the ICP peer. This ﬁeld is optional; if you do not specify the hostname of the ICP peer, you must specify the IP address. host_IP The IP address of the ICP peer. This ﬁeld is optional; if you do not specify the IP address of the ICP peer, you must specify the hostname. ctype Use the following options: • 1 to indicate an ICP parent cache • 2 to indicate an ICP sibling cache • 3 to indicate an ICP local cache proxy_port The port number of the TCP port used by the ICP peer for proxy communication.
3.11. Conﬁguration Files 223 Apache-Trafﬁcserver-Server Documentation, latest icp_port The port number of the UDP port used by the ICP peer for ICP communication. MC_on Enable or disable MultiCast: • 0 if multicast is disabled • 1 if multicast is enabled MC_ip The MultiCast IP address. MC_ttl The multicast time to live. Use the following options: • 1 if IP multicast datagrams will not be forwarded beyond a single subnetwork • 2 to allow delivery of IP multicast datagrams to more than one subnet (if there are one or more multicast routers attached to the ﬁrst hop subnet).
Examples
The following example conﬁguration is for three nodes: the local host, one parent, and one sibling. localhost:0.0.0.0:3:8080:3130:0:0.0.0.0:1 host1:123.12.1.23:1:8080:3131:0:0.0.0.0:1 host2:123.12.1.24:2:8080:3131:0:0.0.0.0:1 ip_allow.conﬁg
The ip_allow.config ﬁle controls client access to the Trafﬁc Server proxy cache. You can specify ranges of IP addresses that are allowed to use the Trafﬁc Server as a web proxy cache. After you modify the ip_allow.config ﬁle, navigate to the Trafﬁc Server bin directory and run the traffic_ctl config reload command to apply changes. When you apply the changes to a node in a cluster, Trafﬁc Server automatically applies the changes to all other nodes in the cluster.
Format
Each line in the ip_allow.config ﬁle must have the following format: src_ip= action= [method=
˓→'|'>] where src_ip is the IP address or range of IP addresses of the client(s). The action ip_allow enables the speciﬁed client(s) to access the Trafﬁc Server proxy cache, and ip_deny denies the speciﬁed client(s) to access the Trafﬁc Server proxy cache. Multiple method keywords can be speciﬁed (method=GET method=HEAD), or multiple methods can be separated by an ‘|’ (method=GET|HEAD). The method keyword is optional and it is defaulted to ALL. This supports ANY string as the HTTP method, meaning no validation is done to check wether it is a valid HTTP method. This allows you to create ﬁlters for any method that your origin may require, this is especially useful if you use newer methods that aren’t know to trafﬁcserver (such as PROPFIND) or if your origin uses an http-ish protocol. By default, the ip_allow.config ﬁle contains the following lines, which allows all methods to localhost to access the Trafﬁc Server proxy cache and denies PUSH, PURGE and DELETE to all IPs (note this allows all other methods to all IPs): src_ip=127.0.0.1 action=ip_allow method=ALL src_ip=::1 action=ip_allow method=ALL src_ip=0.0.0.0-255.255.255.255 action=ip_deny
˓→method=PUSH|PURGE|DELETE src_ip=::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff action=ip_deny
˓→method=PUSH|PURGE|DELETE
224 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Examples
The following example enables all clients to access the Trafﬁc Server proxy cache:
 src_ip=0.0.0.0-255.255.255.255 action=ip_allow
The following example allows all clients on a speciﬁc subnet to access the Trafﬁc Server proxy cache:
 src_ip=123.12.3.000-123.12.3.123 action=ip_allow
The following example denies all clients on a speciﬁc subnet to access the Trafﬁc Server proxy cache:
 src_ip=123.45.6.0-123.45.6.123 action=ip_deny
 log_hosts.conﬁg
To record HTTP transactions for different origin servers in separate log ﬁles, you must list each origin server hostname in the log_hosts.config ﬁle. You should use the same log_hosts.config ﬁle on every Trafﬁc Server node in your cluster. After you modify the log_hosts.config ﬁle, run the traffic_ctl config reload command to apply the changes. When you apply the changes to a node in a cluster, Trafﬁc Server automatically applies the changes to all other nodes in the cluster.
Format
Each line in the log_hosts.config ﬁle has the following format:
 hostname
 where hostname is the hostname of the origin server.
: You can specify keywords in the log_hosts.config ﬁle to record all transactions from origin servers with the speciﬁed keyword in their names in a separate log ﬁle. See the example below.
Examples
The following example conﬁgures Trafﬁc Server to create separate log ﬁles containing all HTTP transactions for the origin servers webserver1, webserver2, and webserver3:
 webserver1 webserver2 webserver3
The following example records all HTTP transactions from origin servers that contain sports in their names. For example: sports.yahoo.com and www.foxsports.com in a log ﬁle called squid-sport.log (the Squid format is enabled):
3.11. Conﬁguration Files 225 Apache-Trafﬁcserver-Server Documentation, latest
 sports
 logs_xml.conﬁg
The logs_xml.config ﬁle deﬁnes the custom log ﬁle formats, ﬁlters, and processing options. The format of this ﬁle is modeled after XML, the Extensible Markup Language.
Format
The logs_xml.config ﬁle contains the speciﬁcations below: • LogFormat speciﬁes the ﬁelds to be gathered from each protocol event access. • LogFilter speciﬁes the ﬁlters that are used to include or exclude certain entries being logged based on the value of a ﬁeld within that entry. • LogObject speciﬁes an object that contains a particular format, a local ﬁlename, ﬁlters, and collation servers. The logs_xml.config ﬁle ignores extra white space, blank lines, and all comments.
LogFormat
The following list shows LogFormat speciﬁcations.  Required Valid format names include any name except squid, common, extended, or extended2, which are pre-deﬁned formats. There is no default for this tag. The format object needs to be above the the LogObject object.  Required A valid format speciﬁcation is a printf-style string describing each log entry when formatted for ASCII output. The printf-style could accept Oct/Hex escape representation: • \abc is Oct escape sequence, a,b,c should be one of [0-9], and (a*8^2 + b*8 + c) should be greater than 0 and less than 255. • \xab is Hex escape sequence, a,b should be one of [0-9, a-f, A-F], and (a*16 + b) should be greater than 0 and less than 255. Use %< field > as a placeholder for valid ﬁeld names. For more information, refer to Custom Logging Fields. The speciﬁed ﬁeld can be one of the following types: Simple. For example: % A ﬁeld within a container, such as an HTTP header or a statistic. Fields of this type have the syntax:
%<{ field } container>
Aggregates, such as COUNT, SUM, AVG, FIRST, LAST. Fields of this type have the syntax: %
: You cannot create a format speciﬁcation that contains both aggregate operators and regular ﬁelds.
 Optional Use this tag when the format contains aggregate operators. The value “aggregate_interval_secs” represents the number of seconds between individual aggregate values being produced.
226 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
The valid set of aggregate operators are: • COUNT • SUM • AVG • FIRST • LAST
LogFilter
The following list shows the LogFilter speciﬁcations.  Required All ﬁlters must be uniquely named.  Required This ﬁeld contains the following elements: valid_log_field - the ﬁeld that will be compared against the given value. For more information, refer to Custom Field Cross-Reference. valid_operator_field - any one of the following: MATCH, CASE_INSENSITIVE_MATCH, CONTAIN, CASE_INSENSITIVE_CONTAIN. • MATCH is true if the ﬁeld and value are identical (case-sensitive). • CASE_INSENSITIVE_MATCH is similar to MATCH, except that it is case-insensitive. • CONTAIN is true if the ﬁeld contains the value (the value is a substring of the ﬁeld). • CASE_INSENSITIVE_CONTAIN is a case-insensitive version of CONTAIN. valid_comparison_value - any string or integer matching the ﬁeld type. For integer values, all of the operators are equivalent and mean that the ﬁeld must be equal to the speciﬁed value. For IP address ﬁelds, this can be a list of IP addresses and include ranges. A range is an IP address, followed by a dash ‘-‘, and then another IP address of the same family. For instance, the 10/8 network can be represented by 10.0.0.0-10.255.255.255. Currently network speciﬁers are not supported.
: There are no negative comparison operators. If you want to specify a negative condition, then use the Action ﬁeld to REJECT the record.
 Required: ACCEPT or REJECT or WIPE_FIELD_VALUE. AC- CEPT or REJECT instructs Trafﬁc Server to either accept or reject records that satisfy the ﬁlter condition. WIPE_FIELD_VALUE wipes out the values of the query params in the url ﬁelds speciﬁed in the Condition. NOTES: 1. WIPE_FIELD_VALUE action is only applied to the parameters in the query part. 2. Multiple parameters can be listed in a single WIPE_FIELD_VALUE ﬁlter 3. If the same parameter appears more than once in the query part , only the value of the ﬁrst occurance is wiped
LogObject
The following list shows the LogObject speciﬁcations.
3.11. Conﬁguration Files 227 Apache-Trafﬁcserver-Server Documentation, latest
 Required Valid format names include the predeﬁned logging formats: squid, common, extended, and extended2, as well as any previously-deﬁned custom log formats. There is no default for this tag. The format object needs to be above the the LogObject object.  Required The ﬁlename to which the given log ﬁle is written on the local ﬁle system or on a remote collation server. No local log ﬁle will be created if you fail to specify this tag. All ﬁlenames are relative to the default logging directory. If the name does not contain an extension (for example, squid), then the extension .log is automatically appended to it for ASCII logs and .blog for binary logs (refer to Mode = “valid_logging_mode”). If you do not want an extension to be added, then end the ﬁlename with a single (.) dot (for example: squid. ).  Optional Valid logging modes include ascii , binary , and ascii_pipe . The default is ascii . • Use ascii to create event log ﬁles in human-readable form (plain ASCII). • Use binary to create event log ﬁles in binary format. Binary log ﬁles generate lower system over- head and occupy less space on the disk (depending on the information being logged). You must use the traffic_logcat utility to translate binary log ﬁles to ASCII format before you can read them. • Use ascii_pipe to write log entries to a UNIX named pipe (a buffer in memory). Other processes can then read the data using standard I/O functions. The advantage of using this option is that Trafﬁc Server does not have to write to disk, which frees disk space and bandwidth for other tasks. In addition, writing to a pipe does not stop when logging space is exhausted because the pipe does not use disk space. If you are using a collation server, then the log is written to a pipe on the collation server. A local pipe is created even before a transaction is processed, so you can see the pipe right after Trafﬁc Server starts. Pipes on a collation server, however, are created when Trafﬁc Server starts.  Optional A comma-separated list of names of any previously-deﬁned log ﬁlters. If more than one ﬁlter is speciﬁed, then all ﬁlters must accept a record for the record to be logged. The ﬁlters need to be above the LogObject object.  Optional A comma-separated list of the protocols this object should log. Valid protocol names for this release are HTTP (FTP is deprecated).  Optional A comma-separated list of valid host- names.This tag indicates that only entries from the named servers will be included in the ﬁle.  Optional A comma-separated list of collation servers (with pipe delimited failover servers) to which all log entries (for this object) are forwarded. Collation servers can be speciﬁed by name or IP address. Specify the collation port with a colon after the name. For example, in host1:5000|failhostA:5000|failhostB:6000, host2:6000 logs would be sent to host1 and host2, with failhostA and failhostB acting as failover hosts for host1. When host1 disconnects, logs would be sent to failhostA. If failhostA disconnects, log entries would be sent to failhostB until host1 or failhostA comes back. Logs would also be sent to host2. 
 Optional The header text you want the log ﬁles to contain. The header text appears at the beginning of the log ﬁle, just before the ﬁrst record.  Optional Enables or disables log ﬁle rolling for the LogObject. This setting overrides the value for the :ts:cv:‘proxy.conﬁg.log.rolling_enabled‘ variable in the records. config ﬁle. Set truth value to one of the following values: • 0 to disable rolling for this particular LogObject. • 1 to roll log ﬁles at speciﬁc intervals during the day (you specify time intervals with the RollingIntervalSec and RollingOffsetHr ﬁelds).
228 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
• 2 to roll log ﬁles when they reach a certain size (you specify the size with the RollingSizeMb ﬁeld). • 3 to roll log ﬁles at speciﬁc intervals during the day or when they reach a certain size (whichever occurs ﬁrst). • 4 to roll log ﬁles at speciﬁc intervals during the day when log ﬁles reach a speciﬁc size (at a speciﬁed time if the ﬁle is of the speciﬁed size).  Optional The seconds between log ﬁle rolling for the LogObject; enables you to specify different rolling intervals for different LogObjects. This setting overrides the value for :ts:cv:‘proxy.conﬁg.log.rolling_interval_sec‘ in the records.config ﬁle.  Optional Speciﬁes an hour (from 0 to 23) at which rolling is guaranteed to align. Rolling might start before then, but a rolled ﬁle will be produced only at that time. The impact of this setting is only noticeable if the rolling interval is larger than one hour. This setting overrides the conﬁguration setting for :ts:cv:‘proxy.conﬁg.log.rolling_offset_hr‘ in the records.config ﬁle.  Optional The size at which log ﬁles are rolled.
Examples
The following is an example of a LogFormat speciﬁcation that collects information using three common ﬁelds:
   :% :%"/> 
The following is an example of a LogFormat speciﬁcation that uses aggregate operators:
   :% :%"/>  
The following is an example of a LogFilter that will cause only REFRESH_HIT entries to be logged:
    MATCH REFRESH_HIT"/> 
: When specifying the ﬁeld in the ﬁlter condition, you can omit the %<>. This means that the ﬁlter below is equivalent to the example directly above:
    
The following is an example of a LogFilter that will cause the value of passwd ﬁeld be wiped in cquc
3.11. Conﬁguration Files 229 Apache-Trafﬁcserver-Server Documentation, latest
    
The following is an example of a LogObject speciﬁcation that creates a local log ﬁle for the minimal format deﬁned earlier. The log ﬁlename will be minimal.log because this is an ASCII log ﬁle (the default).:
   
The following is an example of a LogObject speciﬁcation that includes only HTTP requests served by hosts in the domain company.com or by the speciﬁc server server.somewhere.com. Log entries are sent to port 4000 of the collation host logs.company.com and to port 5000 of the collation host 209.131.52.129.
      
WELF
Trafﬁc Server supports WELF (WebTrends Enhanced Log Format) so you can analyze Trafﬁc Server log ﬁles with WebTrends reporting tools. A predeﬁned  that is compatible with WELF is provided in the logs_xml.config ﬁle (shown below). To create a WELF format log ﬁle, create a  that uses this predeﬁned format.
  % \" fw=% pri=6 proto=% duration=% sent=% rcvd=% src=% dst=% dstname=% user=% op=% arg=\"%\" result=% ref=\"%<{Referer}cqh>\" agent=\"%<{user-agent}cqh>\" cache=%"/> 
 parent.conﬁg
The parent.config ﬁle identiﬁes the parent proxies used in an cache hierarchy. Use this ﬁle to perform the following conﬁguration: • Set up parent cache hierarchies, with multiple parents and parent failover • Conﬁgure selected URL requests to bypass parent proxies Trafﬁc Server uses the parent.config ﬁle only when the parent caching option is enabled (refer to Conﬁguring Trafﬁc Server to Use a Parent Cache).
230 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
After you modify the parent.config ﬁle, run the traffic_ctl config reload command to apply your changes. When you apply the changes to one node in a cluster, Trafﬁc Server automatically applies the changes to all other nodes in the cluster.
Format
Each line in the parent.config ﬁle must contain a parent caching rule. Trafﬁc Server recognizes three space- delimited tags:
 primary_destination=value secondary_specifier=value action=value
The following list shows the possible primary destinations and their allowed values. dest_domain A requested domain name, and its subdomains. dest_host A requested hostname. dest_ip A requested IP address or range of IP addresses separated by a dash (-). url_regex A regular expression (regex) to be found in a URL The secondary speciﬁers are optional in the parent.config ﬁle. The following list shows the possible secondary speciﬁers and their allowed values. port A requested URL port. scheme A request URL protocol: http or https. prefix A preﬁx in the path part of a URL. suffix A ﬁle sufﬁx in the URL. method A request URL method. It can be one of the following: • get • post • put • trace time A time range, such as 08:00-14:00, during which the parent cache is used to serve requests. src_ip A client IP address. internal A boolean value, true or false, specifying if the rule should match (or not match) a transaction originating from an internal API. This is useful to differentiate transaction originating from an ATS plugin. The following list shows the possible actions and their allowed values. parent An ordered list of parent servers. If the request cannot be handled by the last parent server in the list, then it will be routed to the origin server. You can specify either a hostname or an IP address, but; you must specify the port number. secondary_parent An optional ordered list of secondary parent servers. This optional list may only be used when round_robin is set to consistent_hash. If the request cannot be handled by a parent server from the parent list, then the request will be re-tried from a server found in this list using a consistent hash of the url. parent_is_proxy One of the following values: • true - This is the default. The list of parents and secondary parents are proxy cache servers.
3.11. Conﬁguration Files 231 Apache-Trafﬁcserver-Server Documentation, latest
• false - The list of parents and secondary parents are the origin servers go_direct ﬂag is ignored and origins are selected using the speciﬁed round_robin algorithm. The FQDN is removed from the http request line. parent_retry If parent_is_proxy is false, then you may conﬁgure parent_retry for one of the follow- ing values: • simple_retry - If the parent origin server returns a 404 response on a request a new parent is selected and the request is retried. The number of retries is controlled by max_simple_retries which is set to 1 by default. • unavailable_server_retry - If the parent returns a 503 response or if the reponse matches a list of http 5xx responses deﬁned in unavailable_server_retry_responses, the currently selected parent is marked down and a new parent is selected to retry the request. The number of retries is controlled by max_unavailable_server_retries which is set to 1 by default. • both - This enables both simple_retry and unavailable_server_retry as described above. unavailable_server_retry_responses If parent_is_proxy is false and parent_retry is set to either unavailable_server_retry or both, this parameter is a comma separated list of http 5xx re- sponse codes that will invoke the unavailable_server_retry described in the parent_retry sec- tion. By default, unavailable_server_retry_responses is set to 503. max_simple_retries By default the value for max_simple_retries is 1. It may be set to any value in the range 1 to 5. If parent_is_proxy is false and parent_retry is set to simple_retry or both a 404 reponse from a parent origin server will cause the request to be retried using a new parent at most 1 to 5 times as conﬁgured by max_simple_retries. max_unavailable_server_retries By default the value for max_unavailable_server_retries is 1. It may be set to any value in the range 1 to 5. If parent_is_proxy is false and parent_retry is set to unavailable_server_retries or both a 503 reponse by default or any http 5xx response listed in the list unavailable_server_retry_responses from a parent origin server will cause the request to be retried using a new parent after ﬁrst marking the current parent down. The request will be retried at most 1 to 5 times as conﬁgured by max_unavailable_server_retries. round_robin One of the following values: • true - Trafﬁc Server goes through the parent cache list in a round robin-based on client IP address. • strict - Trafﬁc Server machines serve requests strictly in turn. For example: machine proxy1 serves the ﬁrst request, proxy2 serves the second request, and so on. • false - Round robin selection does not occur. • consistent_hash - consistent hash of the url so that one parent is chosen for a given url. If a parent is down, the trafﬁc that would go to the down parent is rehashed amongst the remaining parents. The other trafﬁc is unaffected. Once the downed parent becomes available, the trafﬁc distribution returns to the pre-down state. go_direct One of the following values: • true - requests bypass parent hierarchies and go directly to the origin server. • false - requests do not bypass parent hierarchies. qstring One of the following values: • consider - Use the query string when ﬁnding a parent. • ignore - Do not consider the query string when ﬁnding a parent.
232 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Examples
The following rule conﬁgures a parent cache hierarchy consisting of Trafﬁc Server (which is the child) and two parents, p1.x.com and p2.x.com. Trafﬁc Server forwards the requests it cannot serve to the parent servers p1.x.com and p2.x.com in a round-robin fashion: round_robin=true dest_domain=. method=get parent="p1.x.com:8080; p2.y.com:8080" round_robin=true round_robin=consistent_hash dest_domain=. method=get parent="p1.x.com:8080|1.0; p2.y.com:8080|2.0" round_
˓→robin=consistent_hash
The following rule conﬁgures Trafﬁc Server to route all requests containing the regular expression politics and the path /viewpoint directly to the origin server (bypassing any parent hierarchies): url_regex=politics prefix=/viewpoint go_direct=true Every line in the parent.config ﬁle must contain either a parent= or go_direct= directive. plugin.conﬁg
Description
The plugin.config ﬁle controls run-time loadable plugins available to Trafﬁc Server, as well as their conﬁgura- tion. Plugins listed in this ﬁle are referred to as global plugins because they are always loaded and have global effect. This is in contrast to plugins speciﬁed in remap.config, whose effects are limited to the speciﬁc mapping rules to which they are applied. Each conﬁguration line consists of a path to an .so ﬁle. This path can either be absolute, or relative to the plugin directory (usually /usr/local/libexec/trafficserver). Failure to load a plugin is fatal, and will cause Trafﬁc Server to abort. In general, it is not possible to know whether it is safe for the service to run without a particular plugin, since plugins can have arbitrary effects on caching and authorization policies.
: Plugins should only be listed once. The order in which the plugins are listed is also the order in which they are chained for request processing.
An option list of whitespace-separated arguments may follow the plugin name. These are passed as an argument vec- tor to the plugin’s initialization function, TSPluginInit(). Arguments that begin with the $ character designate Trafﬁc Server conﬁguration variables. These arguments will be replaced with the value of the corresponding conﬁgu- ration variable before the plugin is loaded. When using conﬁguration variable expansion, note that most Trafﬁc Server conﬁguration can be changed. If a plugin requires the current value, it must obtain that using the management API.
Examples
# Comments start with a '#' and continue to the end of the line # Blank lines are ignored # # test-plugin.so arg1 arg2 arg3 # plugins/iwx/iwx.so plugins/abuse/abuse.so etc/trafficserver/abuse.config plugins/icx/icx.so etc/trafficserver/icx.config $proxy.config.http.connect_attempts_
˓→timeout
3.11. Conﬁguration Files 233 Apache-Trafﬁcserver-Server Documentation, latest
See Also
TSAPI(3ts), TSPluginInit(3ts), remap.config(5) records.conﬁg
The records.config ﬁle (by default, located in /usr/local/etc/trafficserver/) is a list of conﬁg- urable variables used by the Trafﬁc Server software. Many of the variables in the records.config ﬁle are set automatically when you set conﬁguration options in Trafﬁc Line. After you modify the records.config ﬁle, run the command traffic_ctl config reload to apply the changes. When you apply changes to one node in a cluster, Trafﬁc Server automatically applies the changes to all other nodes in the cluster.
Format
Each variable has the following format:
SCOPE variable_name DATATYPE variable_value where SCOPE is related to clustering and is either CONFIG (all members of the cluster) or LOCAL (only the local machine) DATATYPE is one of INT (integer), STRING (string), FLOAT (ﬂoating point). A variable marked as Deprecated is still functional but should be avoided as it may be removed in a future release without warning. A variable marked as Reloadable can be updated via the command: traffic_ctl config reload
A variable marked as Overridable can be changed on a per-remap basis using plugins (like the Conﬁguration Remap Plugin). INT type conﬁgurations are expressed as any normal integer, e.g. 32768. They can also be expressed using more human readable values using standard preﬁxes, e.g. 32K. The following preﬁxes are supported for all INT type conﬁgurations • K Kilobytes (1024 bytes) • M Megabytes (1024^2 or 1,048,576 bytes) • G Gigabytes (1024^3 or 1,073,741,824 bytes) • T Terabytes (1024^4 or 1,099,511,627,776 bytes)
: Trafﬁc Server currently writes back conﬁgurations to disk periodically, and when doing so, will not preserve the preﬁxes.
Examples
In the following example, the variable ‘proxy.conﬁg.proxy_name‘_ is a STRING datatype with the value my_server. This means that the name of the Trafﬁc Server proxy is my_server.
234 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
CONFIG proxy.config.proxy_name STRING my_server
If the server name should be that_server the line would be
CONFIG proxy.config.proxy_name STRING that_server
In the following example, the variable proxy.config.arm.enabled is a yes/no ﬂag. A value of 0 (zero) disables the option; a value of 1 enables the option.
CONFIG proxy.config.arm.enabled INT0
In the following example, the variable sets the cluster startup timeout to 10 seconds.
CONFIG proxy.config.cluster.startup_timeout INT 10
The last examples conﬁgures a 64GB RAM cache, using a human readable preﬁx.
CONFIG proxy.config.cache.ram_cache.size INT 64G
Environment Overrides
Every records.config conﬁguration variable can be overridden by a corresponding environment variable. This can be useful in situations where you need a static records.config but still want to tweak one or two settings. The override variable is formed by converting the records.config variable name to upper case, and replacing any dot separators with an underscore. Overriding a variable from the environment is permanent and will not be affected by future conﬁguration changes made in records.config or applied with traffic_ctl. For example, we could override the ‘proxy.conﬁg.product_company‘_ variable like this:
$ PROXY_CONFIG_PRODUCT_COMPANY=example traffic_cop & $ traffic_ctl config get proxy.config.product_company
Conﬁguration Variables
The following list describes the conﬁguration variables available in the records.config ﬁle.
System Variables
A value of 0 means no signal will be sent. Value Effect 0 assign threads to machine 1 assign threads to NUMA nodes [default] 2 assign threads to sockets 3 assign threads to cores 4 assign threads to processing units
: This option only has an affect when Trafﬁc Server has been compiled with --enable-hwloc.
3.11. Conﬁguration Files 235 Apache-Trafﬁcserver-Server Documentation, latest
Metrics Variables
Network
Unless explicitly speciﬁed in ‘proxy.conﬁg.http.server_ports‘_ the server port will be bound to one of these ad- dresses, selected by IP address family. The built in default is any address. This is used if no address for a family is speciﬁed. This setting is useful if most or all server ports should be bound to the same address.
: This is ignored for inbound transparent server ports because they must be able to accept connections on arbitrary IP addresses.
Example
Set the global default for IPv4 to 192.168.101.18 and leave the global default for IPv6 as any address.: LOCAL proxy.local.incoming_ip_to_bind STRING 192.168.101.18
Example
Set the global default for IPv4 to 191.68.101.18 and the global default for IPv6 to fc07:192:168:101::17.: LOCAL proxy.local.incoming_ip_to_bind STRING 192.168.101.18 [fc07:192:168:101::17]
: This is ignored for outbound transparent ports as the local outbound address will be the same as the client local address.
Example
Set the default local outbound IP address for IPv4 connections to 192.168.101.18.: LOCAL proxy.local.outgoing_ip_to_bind STRING 192.168.101.18
Example
Set the default local outbound IP address to 192.168.101.17 for IPv4 and fc07:192:168:101::17 for IPv6.: LOCAL proxy.local.outgoing_ip_to_bind STRING 192.168.101.17 [fc07:192:168:101::17]
236 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Cluster
Value Effect 1 full-clustering mode 2 management-only mode 3 no clustering The network interface to be used for cluster communication. This has to be identical on all members of a clsuter. ToDo: Is that reasonable ?? Should this be local”
Local Manager
This setting speciﬁes whether the management API should be restricted to root processes. If this is set to 0, then on platforms that support passing process credentials, non-root processes will be allowed to make read-only management API calls. Any management API calls that modify server state (eg. setting a conﬁguration variable) will still be restricted to root processes. This setting is not reloadable, since it is must be applied when program:trafﬁc_manager initializes.
Process Manager
Alarm Conﬁguration
HTTP Engine
This is a list, separated by space or comma, of port descriptors. Each descriptor is a sequence of keywords and values separated by colons. Not all keywords have values, those that do are speciﬁcally noted. Keywords with values can have an optional ‘=’ character separating the keyword and value. The case of keywords is ignored. The order of keywords is irrelevant but unspeciﬁed results may occur if incompatible options are used (noted below). Options without values are idempotent. Options with values use the last (right most) value speciﬁed, except for ip-out as detailed later. Quick reference chart. Name Note Deﬁnition number Required The local port. blind Blind (CONNECT) port. compress N/I Compressed. Not implemented. ipv4 Default Bind to IPv4 address family. ipv6 Bind to IPv6 address family. ip-in Value Local inbound IP address. ip-out Value Local outbound IP address. ip-resolve Value IP address resolution style. proto Value List of supported session protocols. ssl SSL terminated. tr-full Fully transparent (inbound and outbound) tr-in Inbound transparent. tr-out Outbound transparent. tr-pass Pass through enabled. number Local IP port to bind. This is the port to which ATS clients will connect. blind Accept only the CONNECT method on this port. Not compatible with: tr-in, ssl. compress Compress the connection. Retained only by inertia, should be considered “not implemented”.
3.11. Conﬁguration Files 237 Apache-Trafﬁcserver-Server Documentation, latest
 ipv4 Use IPv4. This is the default and is included primarily for completeness. This forced if the ip-in option is used with an IPv4 address. ipv6 Use IPv6. This is forced if the ip-in option is used with an IPv6 address. ssl Require SSL termination for inbound connections. SSL must be conﬁgured for this option to provide a functional server port. Not compatible with: blind. proto Specify the session level protocols supported. These should be separated by semi-colons. For TLS proxy ports the default value is all available protocols. For non-TLS proxy ports the default is HTTP only. tr-full Fully transparent. This is a convenience option and is identical to specifying both tr-in and tr-out. Not compatible with: Any option not compatible with tr-in or tr-out. tr-in Inbound transparent. The proxy port will accept connections to any IP address on the port. To have IPv6 inbound transparent you must use this and the ipv6 option. This overrides :ts:cv:‘proxy.local.incoming_ip_to_bind‘ for this port. Not compatible with: ip-in, blind tr-out Outbound transparent. If ATS connects to an origin server for a transaction on this port, it will use the client’s address as its local address. This overrides :ts:cv:‘proxy.local.outgoing_ip_to_bind‘ for this port. Not compatible with: ip-out, ip-resolve tr-pass Transparent pass through. This option is useful only for inbound transparent proxy ports. If the parsing of the expected HTTP header fails, then the transaction is switched to a blind tunnel instead of generating an error response to the client. It effectively enables :ts:cv:‘proxy.conﬁg.http.use_client_target_addr‘ for the transaction as there is no other place to obtain the origin server address. ip-in Set the local IP address for the port. This is the address to which clients will connect. This forces the IP address family for the port. The ipv4 or ipv6 can be used but it is optional and is an error for it to disagree with the IP address family of this value. An IPv6 address must be enclosed in square brackets. If this option is omitted :ts:cv:‘proxy.local.incoming_ip_to_bind‘ is used. Not compatible with: tr-in. ip-out Set the local IP address for outbound connections. This is the address used by ATS locally when it connects to an origin server for transactions on this port. If this is omitted :ts:cv:‘proxy.local.outgoing_ip_to_bind‘ is used. This option can used multiple times, once for each IP address family. The address used is selected by the IP address family of the origin server address. Not compatible with: tr-out. ip-resolve Set the :ts:cv:‘host resolution style ‘ for transactions on this proxy port. Not compatible with: tr-out - this option requires a value of client;none which is forced and should not be explicitly speciﬁed.
Example
Listen on port 80 on any address for IPv4 and IPv6.: 80 80:ipv6
238 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Example
Listen transparently on any IPv4 address on port 8080, and transparently on port 8080 on local address fc01:10:10:1::1 (which implies ipv6).: IPv4:tr-FULL:8080TR-full:IP- in=[fc02:10:10:1::1]:8080
Example
Listen on port 8080 for IPv6, fully transparent. Set up an SSL port on 443. These ports will use the IP address from :ts:cv:‘proxy.local.incoming_ip_to_bind‘. Listen on IP address 192.168.17.1, port 80, IPv4, and connect to origin servers using the local address 10.10.10.1 for IPv4 and fc01:10:10:1::1 for IPv6.: 8080:ipv6:tr-full 443:ssl ip-in=192.168.17.1:80:ip-out=[fc01:10:10:1::1]:ip-out=10.
˓→10.10.1
Example
Listen on port 9090 for TSL enabled HTTP/2 or HTTP connections, accept no other session protocols.: 9090:proto=http2;http:ssl
Trafﬁc Server allows tunnels only to the speciﬁed ports. Supports both wildcards (‘*’) and ranges (“0-1023”).
: These are the ports on the origin server, not Trafﬁc Server :ts:cv:‘proxy ports ‘.
Value Effect 0 Do not modify / set this via header 1 Update the via, with normal verbosity 2 Update the via, with higher verbosity 3 Update the via, with highest verbosity
: The Via header string can be decoded with the Via Decoder Ring.
Value Effect 0 Do not modify / set this via header 1 Update the via, with normal verbosity 2 Update the via, with higher verbosity 3 Update the via, with highest verbosity
: The Via header string can be decoded with the Via Decoder Ring.
Parent Proxy Conﬁguration
HTTP Connection Timeouts
The value of 0 speciﬁes that there is no timeout.
3.11. Conﬁguration Files 239 Apache-Trafﬁcserver-Server Documentation, latest
The default value of 0 speciﬁes that there is no timeout.
HTTP Redirection
Origin Server Connect Attempts
Congestion Control
Negative Response Caching
Proxy User Variables
Security
Cache Control
RAM Cache
Heuristic Expiration
: These fuzzing options are marked as deprecated as of v6.2.0, and will be removed for v7.0.0. Instead, we recommend looking at the new :ts:cv:‘proxy-conﬁg-http-cache-open-write-fail-action‘ conﬁguration and the features around thundering heard avoidance (see cache-basics for details).
Dynamic Content & Content Negotiation
For example: if you specify User-agent, then Trafﬁc Server caches all the different user-agent versions of docu- ments it encounters.
Customizable User Response Pages
DNS
: If the variable :ts:cv:‘proxy.conﬁg.http.enable_url_expandomatic‘ is set to 1 (the default value), then you do not have to add www. and .com to this list because Trafﬁc Server automatically tries www. and .com after trying the values you’ve speciﬁed.
HostDB
: For values above 200000, you must increase :ts:cv:‘proxy.conﬁg.hostdb.storage_size‘ by at least 44 bytes per entry.
240 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
When this and :ts:cv:‘proxy.conﬁg.hostdb.timed_round_robin‘ are both disabled (set to 0), Trafﬁc Server al- ways uses the same origin server for the same client, for as long as the origin server is available. Oth- erwise if this is set then IP address is rotated on every request. This setting takes precedence over :ts:cv:‘proxy.conﬁg.hostdb.timed_round_robin‘. When this and :ts:cv:‘proxy.conﬁg.hostdb.strict_round_robin‘ are both disabled (set to 0), Trafﬁc Server always uses the same origin server for the same client, for as long as the origin server is available. Otherwise if this is set to :arg:‘N‘ the IP address is rotated if more than :arg:‘N‘ seconds have past since the ﬁrst time the current address was used. If this is set (non-empty) then the ﬁle is presumed to be a hosts ﬁle in the standard host ﬁle format. It is read and the entries there added to the HostDB. The ﬁle is periodically checked for a more recent modiﬁcation date in which case it is reloaded. The interval is set by the value :ts:cv:‘proxy.conﬁg.hostdb.host_ﬁle.interval‘. While not technically reloadable, the value is read every time the ﬁle is to be checked so that if changed the new value will be used on the next check and the ﬁle will be treated as modiﬁed. The ﬁle is checked every this many seconds to see if it has changed. If so the HostDB is updated with the new values in the ﬁle. This is an ordered list of keywords separated by semicolons that specify how a host name is to be resolved to an IP address. The keywords are case insensitive. Keyword Meaning ipv4 Resolve to an IPv4 address. ipv6 Resolve to an IPv6 address. client Resolve to the same family as the client IP address. none Stop resolving. The order of the keywords is critical. When a host name needs to be resolved it is resolved in same order as the keywords. If a resolution fails, the next option in the list is tried. The keyword none means to give up resolution entirely. The keyword list has a maximum length of three keywords, more are never needed. By default there is an implicit ipv4;ipv6 attached to the end of the string unless the keyword none appears.
Example
Use the incoming client family, then try IPv4 and IPv6. client;ipv4;ipv6
Because of the implicit resolution this can also be expressed as just client
Example
Resolve only to IPv4. ipv4;none
Example
Resolve only to the same family as the client (do not permit cross family transactions). client;none
3.11. Conﬁguration Files 241 Apache-Trafﬁcserver-Server Documentation, latest
This value is a global default that can be overridden by :ts:cv:‘proxy.conﬁg.http.server_ports‘.
: This style is used as a convenience for the administrator. During a resolution the resolution order will be one family, then possibly the other. This is determined by changing client to ipv4 or ipv6 based on the client IP address and then removing duplicates.
: This option has no effect on outbound transparent connections The local IP address used in the connection to the origin server is determined by the client, which forces the IP address family of the address used for the origin server. In effect, outbound transparent connections always use a resolution style of “client”.
Logging Conﬁguration
: All ﬁles in the logging directory contribute to the space used, even if they are not log ﬁles. In collation client mode, if there is no local disk logging, or :ts:cv:‘proxy.conﬁg.log.max_space_mb_for_orphan_logs‘ is set to a higher value than :ts:cv:‘proxy.conﬁg.log.max_space_mb_for_logs‘, TS will take :ts:cv:‘proxy.conﬁg.log.max_space_mb_for_orphan_logs‘ for maximum allowed log space.
: When max_space_mb_for_orphan_logs is take as the maximum allowed log space in the logging system, the same rule apply to proxy.conﬁg.log.max_space_mb_for_logs also apply to proxy.conﬁg.log.max_space_mb_for_orphan_logs, ie: All ﬁles in the logging directory contribute to the space used, even if they are not log ﬁles. you may need to consider this when you enable full remote logging, and bump to the same size as proxy.conﬁg.log.max_space_mb_for_logs.
: The following defaults (with the format‘‘CONFIG proxy.conﬁg.log.[format].[default]‘‘ have been removed from records.config. They can be accessed by editing logs_xml.config. • log_enabled INT 1 Enables (1) or disables (0) the ﬁle format. • log_is_ascii INT 0 The log ﬁle type: – 1 = ASCII – 0 = binary • log_name STRING [format] The ﬁlename (ex. squid log). • log_header STRING NULL The ﬁle header text (ex. squid log). The format can be either squid (Squid), common (Netscape Common), extended (Netscape Extended), or extended2 (Netscape Extended2).
242 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Value Effect 0 Log collation is disabled. 1 This host is a log collation server. 2 This host is a collation client and sends entries using standard formats to the collation server. 3 This host is a collation client and sends entries using the traditional custom formats to the collation server. 4 This host is a collation client and sends entries that use both the standard and traditional custom formats to the collation server. For information on sending XML-based custom formats to the collation server, refer to Log Formats and logs_xml. config.
: Although Trafﬁc Server supports traditional custom logging, you should use the more versatile XML-based custom formats.
Diagnostic Logging Conﬁguration
Example
To log debug diagnostics to both syslog and diags.log: CONFIG proxy.config.diags.output.debug STRING SL
Tag Subsytem usage dns DNS query resolution http_hdrs Logs the headers for HTTP requests and responses privileges Privilege elevation ssl TLS termination and certiﬁcate processing Trafﬁc Server plugins will typically log debug messages using the TSDebug() API, passing the plugin name as the debug tag.
Reverse Proxy
URL Remap Rules
SSL Termination
Client-Related Conﬁguration
OCSP Stapling Conﬁguration
ICP Conﬁguration
HTTP/2 Conﬁguration
SPDY Conﬁguration
Plug-in Conﬁguration
3.11. Conﬁguration Files 243 Apache-Trafﬁcserver-Server Documentation, latest
SOCKS Processor
Sockets
Value Effect 0 Decouple inbound connections and cache initialization. Connections will be accepted as soon as possible and Trafﬁc Server will run regardless of the results of cache initialization. 1 Do not accept inbound connections until cache initialization has ﬁnished. Trafﬁc server will run regardless of the results of cache initialization. 2 Do not accept inbound connections until cache initialization has ﬁnished and been sufﬁciently successful that cache is enabled. This means at least one cache span is usable. If there are no spans in storage.config or none of the spans can be successfully parsed and initialized then Trafﬁc Server will shut down. 3 Do not accept inbound connections until cache initialization has ﬁnished and been completely successful. This requires at least one cache span in storage.config and that every span speciﬁed is valid and successfully initialized. Any error will cause Trafﬁc Server to shut down.
 remap.conﬁg
The remap.config ﬁle (by default, located in /usr/local/etc/trafficserver/) contains mapping rules that Trafﬁc Server uses to perform the following actions: • Map URL requests for a speciﬁc origin server to the appropriate location on Trafﬁc Server when Trafﬁc Server acts as a reverse proxy for that particular origin server • Reverse-map server location headers so that when origin servers respond to a request with a location header that redirects the client to another location, the clients do not bypass Trafﬁc Server • Redirect HTTP requests permanently or temporarily without Trafﬁc Server having to contact any origin servers Refer to Reverse Proxy and HTTP Redirects, for information about redirecting HTTP requests and using reverse proxy. After you modify the remap.config run the traffic_ctl config reload to apply the changes. When you apply the changes to one node in a cluster, Trafﬁc Server automatically applies the changes to all other nodes in the cluster.
Format
Each line in the remap.config ﬁle must contain a mapping rule. Empty lines, or lines starting with # are ignored. Each line can be broken up into multiple lines for better readability by using \ as continuation marker. Trafﬁc Server recognizes three space-delimited ﬁelds: type, target, and replacement. The following list describes the format of each ﬁeld. type Enter one of the following: • map –translates an incoming request URL to the appropriate origin server URL. • map_with_recv_port –exactly like ‘map’ except that it uses the port at which the request was received to perform the mapping instead of the port present in the request. The regex qualiﬁer can also be used for this type. When present, ‘map_with_recv_port’ mappings are checked ﬁrst. If there is a match, then it is chosen without evaluating the “regular” forward mapping rules. • map_with_referer – extended version of ‘map’, which can be used to activate “deep linking protec- tion”, where target URLs are only accessible when the Referer header is set to a URL that is allowed to link to the target.
244 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
• reverse_map –translates the URL in origin server redirect responses to point to the Trafﬁc Server. • redirect –redirects HTTP requests permanently without having to contact the origin server. Permanent redirects notify the browser of the URL change (by returning an HTTP status code 301) so that the browser can update bookmarks. • redirect_temporary –redirects HTTP requests temporarily without having to contact the origin server. Temporary redirects notify the browser of the URL change for the current request only (by re- turning an HTTP status code 307). target Enter the origin (“from”) URL. You can enter up to four components:
 scheme://host:port/path_prefix
 where scheme is http, https, ws or wss. replacement Enter the origin (“from”) URL. You can enter up to four components:
 scheme://host:port/path_prefix
 where scheme is http, https, ws or wss.
Precedence
Remap rules are not processed top-down, but based on an internal priority. Once these rules are executed we pick the ﬁrst match based on conﬁguration ﬁle parse order. 1. map_with_recv_port and `regex_map_with_recv_port` 2. map and regex_map and reverse_map 3. redirect and redirect_temporary 4. regex_redirect and regex_redirect_temporary
Match-All
A map rule with a single / acts as a wildcard, it will match any request. This should be use with care, and certainly only once at the end of the remap.conﬁg ﬁle. E.g. map/ http://all.example.com
Examples
The following section shows example mapping rules in the remap.config ﬁle.
Reverse Proxy Mapping Rules
The following example shows a map rule that does not specify a path preﬁx in the target or replacement: map http://www.x.com/ http://server.hoster.com/ reverse_map http://server.hoster.com/ http://www.x.com/
3.11. Conﬁguration Files 245 Apache-Trafﬁcserver-Server Documentation, latest
This rule results in the following translations: Client Request Translated Request http://www.x.com/Widgets/index. http://server.hoster.com/Widgets/index. html html http://www.x.com/cgi/form/submit. http://server.hoster.com/cgi/form/ sh?arg=true submit.sh?arg=true The following example shows a map rule with path preﬁxes speciﬁed in the target: map http://www.y.com/marketing/ http://marketing.y.com/ reverse_map http://marketing.y.com/ http://www.y.com/marketing/ map http://www.y.com/sales/ http://sales.y.com/ reverse_map http://sales.y.com/ http://www.y.com/sales/ map http://www.y.com/engineering/ http://engineering.y.com/ reverse_map http://engineering.y.com/ http://www.y.com/engineering/ map http://www.y.com/stuff/ http://info.y.com/ reverse_map http://info.y.com/ http://www.y.com/stuff/
These rules result in the following translations: Client Request Translated Request http://www.y.com/marketing/projects/ http://marketing.y.com/projects/ manhattan/specs.html manhattan/specs.html http://www.y.com/stuff/marketing/ http://info.y.com/marketing/ projects/boston/specs.html projects/boston/specs.html http://www.y.com/engineering/ http://engineering.y.com/marketing/ marketing/requirements.html requirements.html The following example shows that the order of the rules matters: map http://www.g.com/ http://external.g.com/ reverse_map http://external.g.com/ http://www.g.com/ map http://www.g.com/stuff/ http://stuff.g.com/ reverse_map http://stuff.g.com/ http://www.g.com/stuff/
These rules result in the following translation. Client Request Translated Request http://www.g.com/stuff/a.gif http://external.g.com/stuff/a.gif In the above examples, the second rule is never applied because all URLs that match the second rule also match the ﬁrst rule. The ﬁrst rule takes precedence because it appears earlier in the remap.config ﬁle. The following example shows a mapping with a path preﬁx speciﬁed in the target and replacement: map http://www.h.com/a/b/ http://server.h.com/customers/x/y reverse_map http://server.h.com/customers/x/y/ http://www.h.com/a/b/
This rule results in the following translation. Client Request Translated Request http://www.h.com/a/b/c/d/doc. http://server.h.com/customers/x/y/c/d/doc. html html http://www.h.com/a/index.html Translation fails The following example shows reverse-map rules: map http://www.x.com/ http://server.hoster.com/x/ reverse_map http://server.hoster.com/x/ http://www.x.com/
246 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
These rules result in the following translations. Client Request Translated Request http://www.x.com/Widgets http://server.hoster.com/x/Widgets
Client Request Origin Server Header Translated Request http://www.x.com/ http://server.hoster.com/x/ http://www.x.com/ Widgets Widgets/ Widgets/ When acting as a reverse proxy for multiple servers, Trafﬁc Server is unable to route to URLs from older browsers that do not send the Host: header. As a solution, set the variable :ts:cv:‘proxy.conﬁg.header.parse.no_host_url_redirect‘ in the records.config ﬁle to the URL to which Trafﬁc Server will redirect requests without host headers.
Redirect Mapping Rules
The following rule permanently redirects all HTTP requests for www.company.com to www.company2.com: redirect http://www.company.com/ http://www.company2.com/
The following rule temporarily redirects all HTTP requests for www.company1.com to www.company2.com: redirect_temporary http://www.company1.com/ http://www.company2.com/
Regular Expression (regex) Remap Support
Regular expressions can be speciﬁed in remapping rules, with the limitations below: • Only the host ﬁeld can contain a regex; the scheme, port, and other ﬁelds cannot. For path manipulation via regexes, use the regex_remap plugin. • The number of capturing subpatterns is limited to 9. This means that $0 through $9 can be used as subtraction placeholders ($0 will be the entire input string). • The number of substitutions in the expansion string is limited to 10. • There is no regex_ equivalent to reverse_remap, so when using regex_remap you should make sure the reverse path is clear by setting (:ts:cv:‘proxy.conﬁg.url_remap.pristine_host_hdr‘)
Examples regex_map http://x([0-9]+).z.com/ http://real-x$1.z.com/ regex_redirect http://old.(.*).z.com http://new.$1.z.com map_with_referer the format of is the following: map_with_referer client-URL origin-server-URL redirect-URL regex1 [regex2...]
‘redirect-URL’ is a redirection URL speciﬁed according to RFC 2616 and can contain special formatting instructions for run-time modiﬁcations of the resulting redirection URL. All regexes Perl compatible regular expressions, which describes the content of the “Referer” header which must be veriﬁed. In case an actual request does not have “Referer” header or it does not match with referer regular expression, the HTTP request will be redirected to ‘redirect-URL’.
3.11. Conﬁguration Files 247 Apache-Trafﬁcserver-Server Documentation, latest
At least one regular expressions must be speciﬁed in order to activate ‘deep linking protection’. There are limitations for the number of referer regular expression strings - 2048. In order to enable the ‘deep linking protection’ feature in Trafﬁc Server, conﬁgure records.conﬁg with:
CONFIG proxy.config.http.referer_filter INT1
In order to enable run-time formatting for redirect URL, conﬁgure:
CONFIG proxy.config.http.referer_format_redirect INT1
When run-time formatting for redirect-URL was enabled the following format symbols can be used:
%r- to substitute original"Referer" header string %f- to substitute client-URL from 'map_with_referer' record %t- to substitute origin-server-URL from 'map_with_referer' record %o- to substitute request URL to origin server, which was created a the result of a mapping operation
Note: There is a special referer type “~*” that can be used in order to specify that the Referer header is optional in the request. If “~*” referer was used in map_with_referer mapping, only requests with Referer header will be veriﬁed for validity. If the “~” symbol was speciﬁed before referer regular expression, it means that the request with a matching referer header will be redirected to redirectURL. It can be used to create a so-called negative referer list. If “*” was used as a referer regular expression - all referers are allowed. Various combinations of “*” and “~” in a referer list can be used to create different ﬁltering rules.
 map_with_referer Examples
 map_with_referer http://y.foo.bar.com/x/yy/ http://foo.bar.com/x/yy/ http://games. ˓→bar.com/new_games. *\.bar\.com www.bar-friends.com
Explanation: Referer header must be in the request, only ”.*.bar.com” and “www.bar-friends.com” are allowed.
 map_with_referer http://y.foo.bar.com/x/yy/ http://foo.bar.com/x/yy/ http://games. ˓→bar.com/new_games * ~.*\.evil\.com
Explanation: Referer header must be in the request but all referers are allowed except ”.*.evil.com”.
 map_with_referer http://y.foo.bar.com/x/yy/ http://foo.bar.com/x/yy/ http://games. ˓→bar.com/error~ ** ~.*\.evil\.com
Explanation: Referer header is optional. However, if Referer header exists, only request from ”.*.evil.com” will be redirected to redirect-URL.
Plugin Chaining
Plugins can be conﬁgured to be evaluated in a speciﬁc order, passing the results from one in to the next (unless a plugin returns 0, then the “chain” is broken).
Examples
 map http://url/path http://url/path \ @plugin=/etc/traffic_server/config/plugins/plugin1.so @pparam=1 @pparam=2\ @plugin=/etc/traffic_server/config/plugins/plugin2.so @pparam=3
248 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 will pass “1” and “2” to plugin1.so and “3” to plugin2.so. This will pass “1” and “2” to plugin1.so and “3” to plugin2.so
Acl Filters
Acl ﬁlters can be created to control access of speciﬁc remap lines. The markup is very similar to that of ip_allow. config, with slight changes to accomodate remap markup
Examples
 map http://foo.example.com/neverpost http://foo.example.com/neverpost @action=deny
˓→@method=post map http://foo.example.com/onlypost http://foo.example.com/onlypost @action=allow
˓→@method=post
 map http://foo.example.com/ http://foo.example.com/ @action=deny @src_ip=1.2.3.4 map http://foo.example.com/ http://foo.example.com/ @action=allow @src_ip=127.0.0.1 map http://foo.example.com/ http://foo.example.com/ @action=allow @src_ip=10.5.2.1
˓→@in_ip=72.209.23.4
 map http://foo.example.com/ http://foo.example.com/ @action=allow @src_ip=127.0.0.1
˓→@method=post @method=get @method=head
Note that these Acl ﬁlters will return a 403 response if the resource is restricted. The difference between @src_ip and @in_ip is that the @src_ip is the client ip and the in_ip is the ip address the client is connecting to (the incoming address).
Named Filters
Named ﬁlters can be created and applied to blocks of mappings using the .definefilter, .activatefilter, and .deactivatefilter directives. Named ﬁlters must be deﬁned using .definefilter before being used. Once deﬁned, .activatefilter can used to activate a ﬁlter for all mappings that follow until deactivated with .deactivatefilter. The @internal operator can be used to ﬁlter on whether a request is generated by Trafﬁc Server itself, usually by a plugin. This operator is helpful for remapping internal requests without allowing access to external users. By default both internal and external requests are allowed.
Examples
.definefilter disable_delete_purge @action=deny @method=delete @method=purge .definefilter local_only @action=allow @src_ip=192.168.0.1-192.168.0.254 @src_ip=10.0.
˓→0.1-10.0.0.254
.activatefilter disable_delete_purge
 map http://foo.example.com/ http://bar.example.com/
.activatefilter local_only map http://www.example.com/admin http://internal.example.com/admin
3.11. Conﬁguration Files 249 Apache-Trafﬁcserver-Server Documentation, latest
.deactivatefilter local_only
 map http://www.example.com/ http://internal.example.com/ map http://auth.example.com/ http://auth.internal.example.com/ @action=allow @internal
The ﬁlter disable_delete_purge will be applied to all of the mapping rules. (It is activated before any mappings and is never deactivated.) The ﬁlter local_only will only be applied to the second mapping.
Including Additional Remap Files
The .include directive allows mapping rules to be spread across multiple ﬁles. The argument to the .include directive is a list of ﬁle names to be parsed for additional mapping rules. Unless the names are absolute paths, they are resolved relative to the Trafﬁc Server conﬁguration directory. The effect of the .include directive is as if the contents of the listed ﬁles is included in the parent and parsing restarted at the point of inclusion. This means that and ﬁlters named in the included ﬁles are global in scope, and that additional .include directives are allowed.
: Included remap ﬁles are not currently tracked by the conﬁguration subsystem. Changes to included remap ﬁles will not be noticed by online conﬁguration changes applied by traffic_ctl config reload unless remap. config has also changed.
Examples
In this example, a top-level remap.config ﬁle simply references additional mapping rules ﬁles
.include filters.config .include one.example.com.config two.example.com.config
The ﬁle ﬁlters.conﬁg contains
.definefilter deny_purge @action=deny @method=purge .definefilter allow_purge @action=allow @method=purge
The ﬁle one.example.com.conﬁg contains:
.activatefilter deny_purge map http://one.example.com http://origin-one.example.com .deactivatefilter deny_purge
The ﬁle two.example.com.conﬁg contains:
.activatefilter allow_purge map http://two.example.com http://origin-two.example.com .deactivatefilter dallowpurge
 splitdns.conﬁg
The splitdns.config ﬁle enables you to specify the DNS server that Trafﬁc Server should use for resolving hosts under speciﬁc conditions. For more information, refer to Split DNS. To specify a DNS server, you must supply the following information in each active line within the ﬁle:
250 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
• A primary destination speciﬁer in the form of a destination domain, a destination host, or a URL regular expres- sion • A set of server directives, listing one or more DNS servers with corresponding port numbers You can also include the following optional information with each DNS server speciﬁcation: • A default domain for resolving hosts • A search list specifying the domain search order when multiple domains are speciﬁed After you modify the splitdns.config ﬁle, run the traffic_ctl config reload command to apply the changes. When you apply changes to a node in a cluster, Trafﬁc Server automatically applies the changes to all other nodes in the cluster.
Format
Each line in the splitdns.config ﬁle uses one of the following formats:
 dest_domain=dest_domain| dest_host| url_regex named=dns_server def_domain=def_
˓→domain search_list=search_list
The following list describes each ﬁeld. dest_domain A valid domain name. This speciﬁes that DNS server selection will be based on the destination domain. You can preﬁx the domain with an exclamation mark (!) to indicate the NOT logical operator. dest_host A valid hostname. This speciﬁes that DNS server selection will be based on the destination host. You can preﬁx the host with an exclamation mark (!) to indicate the NOT logical operator. url_regex A valid URL regular expression. This speciﬁes that DNS server selection will be based on a regular expression. named This is a required directive. It identiﬁes the DNS server that Trafﬁc Server should use with the given destina- tion speciﬁer. You can specify a port using a colon (:). If you do not specify a port, then 53 is used. Specify multiple DNS servers with spaces or semicolons (;) as separators. You must specify the domains with IP addresses in CIDR (“dot”) notation. def_domain A valid domain name. This optional directive speciﬁes the default domain name to use for resolving hosts. Only one entry is allowed. If you do not provide the default domain, the system determines its value from /etc/resolv.conf search_list A list of domains separated by spaces or semicolons (;). This speciﬁes the domain search order. If you do not provide the search list, the system determines the value from resolv.conf(5)
Examples
Consider the following DNS server selection speciﬁcations:
 dest_domain=internal.company.com named=255.255.255.255:212 255.255.255.254 def_
˓→domain=company.com search_list=company.com company1.com dest_domain=!internal.company.com named=255.255.255.253
Now consider the following two requests:
 http://minstar.internal.company.com
3.11. Conﬁguration Files 251 Apache-Trafﬁcserver-Server Documentation, latest
This request matches the ﬁrst line and therefore selects DNS server 255.255.255.255 on port 212. All resolver requests use company.com as the default domain, and company.com and company1.com as the set of domains to search ﬁrst.
 http://www.microsoft.com
This request matches the second line. Therefore, Trafﬁc Server selects DNS server 255.255.255.253. Because no def_domain or search_list was supplied, Trafﬁc Server retrieves this information from resolv.conf(5)
 ssl_multicert.conﬁg
The ssl_multicert.config ﬁle lets you conﬁgure Trafﬁc Server to use multiple SSL server certiﬁcates to terminate the SSL sessions. If you have a Trafﬁc Server system with more than one IP address assigned to it, then you can assign a different SSL certiﬁcate to be served when a client requests a particular IP address or host name. At conﬁguration time, certiﬁcates are parsed to extract the certiﬁcate subject and all the DNS subject alternative names. A certiﬁcate will be presented for connections requesting any of the hostnames found in the certiﬁcate. Wildcard names are supported, but only of the form *.domain.com, ie. where * is the leftmost domain component. Changes to ssl_multicert.config can be applied to a running Trafﬁc Server using traffic_ctl config reload.
Format
Each ssl_multicert.config line consists of a sequence of key=value ﬁelds that specify how Trafﬁc Server should use a particular SSL certiﬁcate. ssl_cert_name=FILENAME[,FILENAME ...] The name of the ﬁle containing the TLS certiﬁcate. FILENAME is located relative to the directory speciﬁed by the :ts:cv:‘proxy.conﬁg.ssl.server.cert.path‘ conﬁguration vari- able. It may also include the intermediate CA certiﬁcates, sorted from leaf to root. At a minimum, the ﬁle must include a leaf certiﬁcate. When running with OpenSSL 1.0.2 or later, this directive can be used to conﬁgure the intermediate CA chain on a per-certiﬁcate basis. Multiple chain ﬁles are separated by comma character. For example, it is possible able to conﬁgure a ECDSA certiﬁcate chain and a RSA certiﬁcate chain and serve them simultaneously, allowing OpenSSL to determine which certiﬁcate would be used when the TLS session cipher suites are negotiated. Note that the leaf certs in FILENAME1 and FILENAME2 must have the same subjects and alternate names. The ﬁrst certiﬁcate is used to to match the client’s SNI request. You can also conﬁgure multiple leaf certiﬁcates in a same chain with OpenSSL 1.0.1. This is the only ﬁeld that is required to be present. dest_ip=ADDRESS (optional) The IP (v4 or v6) address that the certiﬁcate should be presented on. This is now only used as a fallback in the case that the TLS SubjectNameIndication extension is not supported. If ADDRESS is *, the corresponding certiﬁcate will be used as the global default fallback if no other match can be made. The address may contain a port speciﬁer, in which case the corresponding certiﬁcate will only match for connections accepted on the speciﬁed port. IPv6 addresses must be enclosed by square brackets if they have a port, eg, [::1]:80. Care should be taken to make each ADDRESS unique. ssl_key_name=FILENAME (optional) The name of the ﬁle containing the private key for this certiﬁcate. If the key is contained in the certiﬁcate ﬁle, this ﬁeld can be omitted, otherwise FILENAME is resolved relative to the :ts:cv:‘proxy.conﬁg.ssl.server.private_key.path‘ conﬁguration variable. ssl_ca_name=FILENAME (optional) If the certiﬁcate is issued by an authority that is not in the system CA bundle, additional certiﬁcates may be needed to validate the certiﬁcate chain. FILENAME is resolved relative to the :ts:cv:‘proxy.conﬁg.ssl.CA.cert.path‘ conﬁguration variable.
252 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest ssl_ticket_enabled=1|0 (optional) Enable RFC 5077 stateless TLS session tickets. To support this, OpenSSL should be upgraded to version 0.9.8f or higher. This option must be set to 0 to disable session ticket support. ticket_key_name=FILENAME (optional) The name of session ticket key ﬁle which contains a secret for encrypt- ing and decrypting TLS session tickets. If FILENAME is not an absolute path, it is resolved relative to the :ts:cv:‘proxy.conﬁg.ssl.server.cert.path‘ conﬁguration variable. This option has no effect if session tick- ets are disabled by the ssl_ticket_enabled option. The contents of the key ﬁle should be 48 random (ASCII) bytes. One way to generate this would be to run head -c48 /dev/urandom | openssl enc -base64 | head -c48 > file.ticket. Session ticket support is enabled by default. If neither of the ssl_ticket_enabled and ticket_key_name options are speciﬁed, and internal session ticket key is generated. This key will be different each time Trafﬁc Server is started. ssl_key_dialog=builtin|”exec:/path/to/program [args]” (optional) Method used to provide a pass phrase for en- crypted private keys. If the pass phrase is incorrect, SSL negotiation for this dest_ip will fail for clients who attempt to connect. Two options are supported: builtin and exec: builtin - Requests pass phrase via stdin/stdout. User will be provided the ssl_cert_name and be prompted for the pass phrase. Useful for debugging. exec: - Executes program /path/to/program and passes args, if speciﬁed, to the program and reads the output from stdout for the pass phrase. If args are provided then the entire exec: string must be quoted with “” (see examples). Arguments with white space are supported by single quoting (‘). The intent is that this program runs a security check to ensure that the system is not compromised by an attacker before providing the pass phrase.
Certiﬁcate Selection
Trafﬁc Server attempts two certiﬁcate selections during SSL connection setup. An initial selection is made when a TCP connection is accepted. This selection examines the IP address and port that the client is connecting to and chooses the best certiﬁcate from the those that have a dest_ip speciﬁcation. If no matching certiﬁcates are found, a default certiﬁcate is chosen. The ﬁnal certiﬁcate selection is made during the SSL handshake. At this point, the client may use Server Name Indication to request a speciﬁc hostname. Trafﬁc Server will use this request to select a certiﬁcate with a matching subject or subject alternative name. Failing that, a wildcard certiﬁcate match is attempted. If no match can be made, the initial certiﬁcate selection remains in force. In all cases, Trafﬁc Server attempts to select the most speciﬁc match. An address speciﬁcation that contains a port number will take precedence over a speciﬁcation that does not contain a port number. A speciﬁc certiﬁcate subject will take precedence over a wildcard certiﬁcate. In the case of multiple matching certiﬁcates the ﬁrst match will be returned to non-SNI capable clients.
Examples
The following example conﬁgures Trafﬁc Server to use the SSL certiﬁcate server.pem for all requests to the IP address 111.11.11.1 and the SSL certiﬁcate server1.pem for all requests to the IP address 11.1.1.1. Connections from all other IP addresses are terminated with the default.pem certiﬁcate. Since the private key is included in the certiﬁcate ﬁles, no private key name is speciﬁed. dest_ip=111.11.11.1 ssl_cert_name=server.pem dest_ip=11.1.1.1 ssl_cert_name=server1.pem dest_ip=* ssl_cert_name=default.pem
The following example conﬁgures Trafﬁc Server to use the ECDSA certiﬁcate chain ecdsa.pem or RSA certiﬁcate chain rsa.pem for all requests.
3.11. Conﬁguration Files 253 Apache-Trafﬁcserver-Server Documentation, latest
 dest_ip=* ssl_cert_name=ecdsa.pem,rsa.pem
The following example conﬁgures Trafﬁc Server to use the ECDSA certiﬁcate chain ecdsa.pem or RSA certiﬁcate chain rsa.pem for all requests, the public key and private key are in separate PEM ﬁles. Note that the number of ﬁles in ssl_key_name must match the ﬁles in ssl_cert_name, and they should be presented in the same order.
 dest_ip=* ssl_cert_name=ecdsa_pub.pem,rsa_pub.pem ssl_key_name=ecdsa_private.pem,rsa_ ˓→private.pem
The following example conﬁgures Trafﬁc Server to use the SSL certiﬁcate server.pem and the private key serverKey.pem for all requests to port 8443 on IP address 111.11.11.1. The general.pem certiﬁcate is used for server name matches.
 dest_ip=111.11.11.1:8443 ssl_cert_name=server.pem ssl_key_name=serverKey.pem ssl_cert_
˓→name=general.pem
The following example conﬁgures Trafﬁc Server to use the SSL certiﬁcate server.pem for all requests to the IP address 111.11.11.1. Session tickets are enabled with a persistent ticket key.
 dest_ip=111.11.11.1 ssl_cert_name=server.pem ssl_ticket_enabled=1 ticket_key_
˓→name=ticket.key
The following example conﬁgures Trafﬁc Server to use the SSL certiﬁcate server.pem and disable session tickets for all requests to the IP address 111.11.11.1.
 dest_ip=111.11.11.1 ssl_cert_name=server.pem ssl_ticket_enabled=0
The following examples conﬁgure Trafﬁc Server to use the SSL certiﬁcate server.pem which includes an encrypted private key. The external program /usr/bin/mypass will be called on startup with one parameter (foo) in the ﬁrst example, and with two parameters (foo) and (ba r) in the second example, the program (mypass) will return the pass phrase to decrypt the keys.
 ssl_cert_name=server1.pem ssl_key_dialog="exec:/usr/bin/mypass foo" ssl_cert_name=server2.pem ssl_key_dialog="exec:/usr/bin/mypass foo'ba r'"
 storage.conﬁg
The storage.config ﬁle (by default, located in /usr/local/etc/trafficserver/) lists all the ﬁles, directories, and/or hard disk partitions that make up the Trafﬁc Server cache. After you modify the storage. config ﬁle the new settings will not be effective until Trafﬁc Server is restarted.
Format
The format of the storage.config ﬁle is a series of lines of the form pathname size [ volume=number ][ id=string ] where :arg:‘pathname‘ is the name of a partition, directory or ﬁle, :arg:‘size‘ is the size of the named partition, direc- tory or ﬁle (in bytes), and :arg:‘volume‘ is the volume number used in the ﬁles volume.config and hosting. config. :arg:‘id‘ is used for seeding the Assignment Table. You must specify a size for directories; size is optional for ﬁles and raw partitions. :arg:‘volume‘ and arg:seed are optional.
254 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
: The :arg:‘volume‘ option is independent of the :arg:‘seed‘ option and either can be used with or without the other, and their ordering on the line is irrelevant.
: If the :arg:‘id‘ option is used every use must have a unique value for :arg:‘string‘.
You can use any partition of any size. For best performance: • Use raw disk partitions. • For each disk, make all partitions the same size. • For each node, use the same number of partitions on all disks. • Group similar kinds of storage into different volumes. For example split out SSD’s or RAM drives into their own volume. Specify pathnames according to your operating system requirements. See the following examples. In the storage. config ﬁle, a formatted or raw disk must be at least 128 MB. When using raw disk or partitions, you should make sure the :ts:cv:‘Trafﬁc Server user ‘ used by the Trafﬁc Server process has read and write privileges on the raw disk device or partition. One good practice is to make sure the device ﬁle is set with ‘g+rw’ and the Trafﬁc Server user is in the group which owns the device ﬁle. However, some operating systems have stronger requirements - see the following examples for more information. As with standard records.config integers, human readable preﬁxes are also supported. They include • K Kilobytes (1024 bytes) • M Megabytes (1024^2 or 1,048,576 bytes) • G Gigabytes (1024^3 or 1,073,741,824 bytes) • T Terabytes (1024^4 or 1,099,511,627,776 bytes)
Assignment Table
Each storage element deﬁned in storage.config is divided in to stripes. The assignment table maps from an object URL to a speciﬁc stripe. The table is initialized based on a pseudo-random process which is seeded by hashing a string for each stripe. This string is composed of a base string, an offset (the start of the stripe on the storage element), and the length of the stripe. By default the path for the storage is used as the base string. This ensures that each stripe has a unique string for the assignment hash. This does make the assignment table very sensitive to the path for the storage elements and changing even one can have a cascading effect which will effectively clear most of the cache. This can be problem when drives fail and a system reboot causes the path names to change. The :arg:‘id‘ option can be used to create a ﬁxed string that an administrator can use to keep the assignment table consistent by maintaing the mapping from physical device to base string even in the presence of hardware changes and failures.
Examples
The following basic example shows 128 MB of cache storage in the /big_dir directory:
/big_dir 134217728
3.11. Conﬁguration Files 255 Apache-Trafﬁcserver-Server Documentation, latest
You can use the . symbol for the current directory. Here is an example for 64 MB of cache storage in the current directory:
. 134217728
As an alternative, using the human readable preﬁxes, you can express a 64GB cache ﬁle with:
/really_big_dir 64G
: When using on-ﬁlesystem cache disk storage, you can only have one such directory speciﬁed. This will be address in a future version.
Solaris Example
The following example is for the Solaris operating system:
/dev/rdsk/c0t0d0s5 /dev/rdsk/c0t0d1s5
: Size is optional. If not speciﬁed, the entire partition is used.
Linux Example
: Rather than refer to disk devices like /dev/sda, /dev/sdb, etc., modern Linux supports alternative symlinked names for disk devices in the /dev/disk directory structure. As noted for the Assignment Table the path used for the disk can effect the cache if it changes. This can be ameloriated in some cases by using one of the alternate paths in via /dev/disk. Note that if the by-id or by-path style is used, replacing a failed drive will cause that path to change because the new drive will have a different physical ID or path. The original hash string can be kept by adding :arg:‘id‘ or :arg:‘path‘ with the original path to the storage line. If this is not sufﬁcient then the :arg:‘id‘ or :arg:‘path‘ argument should be used to create a more permanent assign- ment table. An example would be:
/dev/sde id=cache.disk.0 /dev/sdg id=cache.disk.1
The following example will use an entire raw disk in the Linux operating system:
/dev/disk/by-id/[DiskA_ID] volume=1 /dev/disk/by-path/[DiskB_Path] volume=2
In order to make sure traffic_server will have access to this disk you can use udev(7) to persistently set the right permissions. The following rules are targeted for an Ubuntu system, and stored in /etc/udev/rules.d/ 51-cache-disk.rules:
# Assign DiskA and DiskB to the tserver group # make the assignment final, no later changes allowed to the group! SUBSYSTEM=="block", KERNEL=="sd[ef]", GROUP:="tserver"
256 Chapter 3. Administrator’s Guide Apache-Trafﬁcserver-Server Documentation, latest
In order to apply these settings, trigger a reload with udevadm(8):: udevadm trigger--subsystem-match=block
FreeBSD Example
Starting with 5.1 FreeBSD dropped support for explicit raw devices. All devices on FreeBSD can be accessed raw now. The following example will use an entire raw disk in the FreeBSD operating system:
/dev/ada1 /dev/ada2
In order to make sure traffic_server will have access to this disk you can use devfs(8) to persistently set the right permissions. The following rules are stored in devfs.conf(5):
# Assign /dev/ada1 and /dev/ada2 to the tserver user own ada[12] tserver:tserver volume.conﬁg
The volume.config ﬁle enables you to manage your cache space more efﬁciently and restrict disk usage by creating cache volumes of different sizes for speciﬁc protocols. You can further conﬁgure these volumes to store data from certain origin servers and/or domains in the hosting.config ﬁle.
: The volume conﬁguration must be the same on all nodes in a cluster. You must stop Trafﬁc Server before you change the cache volume size and protocol assignment. For step-by-step instructions about partitioning the cache, refer to Partitioning the Cache.
Format
For each volume you want to create, enter a line with the following format: volume=volume_number scheme=protocol_type size=volume_size where volume_number is a number between 1 and 255 (the maximum number of volumes is 255) and protocol_type is http. Trafﬁc Server supports http for HTTP volume types; volume_size is the amount of cache space allocated to the volume. This value can be either a percentage of the total cache space or an absolute value. The absolute value must be a multiple of 128 MB, where 128 MB is the smallest value. If you specify a percentage, then the size is rounded down to the closest multiple of 128 MB. Each volume is striped across several disks to achieve parallel I/O. For example: if there are four disks, then a 1-GB volume will have 256 MB on each disk (assuming each disk has enough free space available). If you do not allocate all the disk space in the cache, then the extra disk space is not used. You can use the extra space later to create new volumes without deleting and clearing the existing volumes.
Examples
The following example partitions the cache evenly between HTTP and HTTPS requests:
3.11. Conﬁguration Files 257 Apache-Trafﬁcserver-Server Documentation, latest
 volume=1 scheme=http size=50% volume=2 scheme=https size=50%
Audience
This section of the Apache Trafﬁc Server™ Manual is aimed at those individuals looking to install and conﬁgure Trafﬁc Server for use as a caching proxy server. The section assumes basic knowledge of the command line and system administration tasks. Detailed steps are provided on how to build and conﬁgure the software, so even if you have never used a compiler or looked at a Makeﬁle before, you will ﬁnd everything you need in the following chapters to get you up and running with your own instance of Trafﬁc Server. Or so is our hope. Should you encounter problems, or ﬁnd any of the instructions unclear or lacking, please seek help through one of the avenues listed in Other Resources.
258 Chapter 3. Administrator’s Guide CHAPTER 4
Developer’s Guide
This documentation is a work in progress. It was originally written for a previous, commercially-available version of Trafﬁc Server that supported different operating systems and more functions than the current version. As a result, some of the sections may refer to functionality that no longer exists. If you ﬁnd any such issues, you may want to submit a bug or a patch. We also have a Wiki page explaining how to create useful bug reports. We encourage everyone to ﬁle tickets, early and often. Looking for existing, duplicate bugs is encouraged, but not required.
Introduction
This guide has the following basic components: • Introduction and overview. • Tutorials about writing speciﬁc kinds of plugins: HTTP header-based plugins, content transformation plugins, and protocol plugins. • Guides about speciﬁc interfaces. • Reference material. If you’re new to writing Trafﬁc Server plugins, Getting Started should be your starting point. Header-Based Plugin Examples provides details about plugins that work on HTTP headers, while HTTP Transformations explains how to write a plugin that transforms or scans the body of an HTTP response. New Protocol Plugins provides essential information if you want to support your own protocol on Trafﬁc Server. For a reference to the C API functions and types that your plugin will use, refer to the API Reference. Below is a section-by-section breakdown of this guide: Getting Started How to compile and load plugins. Walks through a simple “hello world” example; explains how to initialize and register plugins. Basic structures that all plugins use: events, continuations, and how to hook on to Trafﬁc Server processes. Detailed explication of a sample blacklisting plugin.
259 Apache-Trafﬁcserver-Server Documentation, latest
Query Remap Plugin This chapter demonstrates on a practical example how you can exploit the Trafﬁc Server remap API for your plugins. Header-Based Plugin Examples Detailed explanation about writing plugins that work on HTTP headers; discusses sample blacklisting and basic authorization plugins. HTTP Transformations Detailed explanation of the null-transform example; also discusses VConnections, VIOs, and IO buffers. New Protocol Plugins Detailed explanation of a sample protocol plugin that supports a synthetic protocol. Discusses VConnections and mutexes, as well as the new NetConnection, DNS lookup, logging, and cache APIs. The remaining sections comprise the API function reference and are organized by function type: Plugin Interfaces Details error-writing and tracing functions, thread functions, and Trafﬁc Server API versions of the malloc and fopen families. The Trafﬁc Server API versions overcome various C library limitations. Hooks and Transactions Functions in this chapter hook your plugin to Trafﬁc Server HTTP processes. HTTP Headers Contains instructions for implementing performance enhancements for all plugins that manipulate HTTP headers. These functions examine and modify HTTP headers, MIME headers, URLs, and the marshal buffers that contain header information. If you are working with headers, then be sure to read this chapter. Mutexes Continuations Continuations provide the basic callback mechanism and data abstractions used in Trafﬁc Server. Plugin Conﬁguration Actions Describes how to use TSActions and the TSDNSLookup API. IO Describes how to use the Trafﬁc Server IO interfaces: TSVConnection, TSVIO, TSIOBuffer, TSNetVConnection, the Cache API. Plugin Management These functions enable you to set up a conﬁguration interface for plugins, access installed plugin ﬁles, and set up plugin licensing. Adding Statistics These functions add statistics to your plugin. API Functions Trafﬁc Server API Function Documentation.
Release Process
Managing a release is easiest in an environment that is as clean as possible. For this reason, cloning the code base in to a new directory for the release process is recommended.
Requirements
• A system for git and building. • A cryptographic key that has been signed by at least two other PMC members. This should be preferentially associated with your apache.org email address but that is not required.
Release Candidate
The ﬁrst step in a release is making a release candidate. This is distributed to the community for validation before the actual release.
260 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Document
Gather up information about the changes for the release. The CHANGES ﬁle is a good starting point. You may also want to check the commits since the last release. The primary purpose of this is to generate a list of the important changes since the last release. Create or update a page on the Wiki for the release. If it is a major or minor release it should have its own page. Use the previous release page as a template. Point releases should get a section at the end of the corresponding release page. Write an announcement for the release. This will contain much of the same information that is on the Wiki page but more concisely. Check the mailing list archives for examples to use as a base.
Build
1. Go to the top level source directory. 2. Check the version in configure.ac. There are two values near the top that need to be set, TS_VERSION_S and TS_VERSION_N. These are the release version number in different encodings. 3. Check the variable RC in the top level Makefile.am. This should be the point release value. This needs to be changed for every release candidate. The ﬁrst release candidate is 0 (zero). 4. Execute the following commands to make the distribution ﬁles.
 autoreconf-i ./configure make rel-candidate
These steps will create the distribution ﬁles and sign them using your key. Expect to be prompted twice for your passphrase unless you use an ssh key agent. If you have multiple keys you will need to set the default appropriately beforehand, as no option will be provided to select the signing key. The ﬁles should have names that start with trafficserver-X.Y.Z-rcA.tar.bz2 where X.Y.Z is the version and A is the release candidate counter. There should be four such ﬁles, one with no extension and three others with the extensions asc, md5, and sha1. This will also create a signed git tag of the form X.Y.Z-rcA.
Distribute
The release candidate ﬁles should be uploaded to some public storage. Your personal storage on people.apach.org is a reasonable location to use. Send the release candiate announcement to the users and dev mailinging lists, noting that it is a release candidate and providing a link to the distribution ﬁles you uploaded. This announcement should also call for a vote on the candidate, generally with a 72 hours time limit. If the voting was successful (at least three “+1” votes and no “-1” votes), proceed to Ofﬁcial Release. Otherwise, repeat the Release Candidate process.
Ofﬁcial Release
Build the distribution ﬁles with the command make release
4.2. Release Process 261 Apache-Trafﬁcserver-Server Documentation, latest
Be sure to not have changed anything since the release candidate was built so the checksums are identical. This will create a signed git tag of the form X.Y.Z and produce the distribution ﬁles. Push the tag to the ASF repository with the command
 git push origin X.Y.Z
This presumes origin is the name for the ASF remote repository which is correct if you originally clone from the ASF repository. The distribution ﬁles must be added to an SVN repository. This can be accessed with the command:
 svn co https://dist.apache.org/repos/dist/release/trafficserver
All four of the distribution ﬁles go here. If you are making a point release then you should also remove the distribution ﬁles for the previous release. Allow 24 hours for the ﬁles to be distributed through the ASF infrastructure. The Trafﬁc Server website must be updated. This is an SVN repository which you can access with
 svn co https://svn.apache.org/repos/asf/trafficserver/site/trunk
The ﬁles of interest are in the content directory. index.html This is the front page. The places to edit here are any security announcements at the top and the “News” section. downloads.en.mdtext Update the downloads page to point to the new download objects. After making changes, commit them and then run
 publish.pl trafficserver
On the people.apache.org host. If needed, update the Wiki page for the release to point at the release distribution ﬁles. Update the announcement, if needed, to refer to the release distribution ﬁles and remove the comments concerning the release candidate. This announcement should be sent to the users and dev mailing lists. It should also be sent to the ASF announcement list, which must be done using an apache.org email address. Finally, update various ﬁles after the release: • The STATUS ﬁle for master and for the release branch to include this version. • The CHANGES ﬁle to have a header for the next version. • configure.ac to be set to the next version. • In the top level Makefile.am change RC to have the value 0.
Contributing to Trafﬁc Server
Using Vagrant to Test Trafﬁc Server
The Apache Trafﬁc Server™ project’s ofﬁcial repository includes a Vagrantﬁle, intended to ease the process of cre- ating environments suitable for building and testing Trafﬁc Server, where all the necessary dependencies are installed automatically for a variety of operating systems and common distribution releases.
262 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Vagrant is a tool for building complete development environments. With an easy-to-use workﬂow and focus on automation, Vagrant lowers development environment setup time, increases develop- ment/production parity, and makes the “works on my machine” excuse a relic of the past. —VagrantUp website Vagrant can be used in combination with any of the popular conﬁgurtion management and automation tools, such as Chef, Puppet, Ansible, and more. The Vagrantﬁle included in the Trafﬁc Server repository happens to make use of Puppet.
Installing Vagrant and Dependencies
VirtualBox
The virtualization software VirtualBox is required to create and run the virtual machines created by the included project Vagrantﬁle. VirtualBox can be obtained by free from the ofﬁcial website, and many distributions provide their own packages as well. No special conﬁguration of the software is required.
Vagrant
A fairly recent version of Vagrant is necessary to use the included Vagrantﬁle. While older versions of Vagrant could be installed through the Ruby Gems facility, modern versions are only provided as distribution speciﬁc packages.
NFS Server
The project Vagrantﬁle uses the NFS shared folders support of VirtualBox to mount the same directory in which the Vagrantﬁle is located on your host machine as a network directory inside the virtual machine. For this to work, your host machine must have an NFS server installed and running, and the user under which you run the vagrant commands must have sudo permissions to modify the NFS exports conﬁguration and restart the NFS service. The virtual machine created by Vagrant will still function without a working NFS server on your host machine, but you will not be able to access the shared folder which includes the entire Trafﬁc Server source tree. You may opt to modify the Vagrantﬁle to use a method other than NFS, as per the Vagrant documentation.
Managing Virtual Machines
Listing Available Machines
The included Vagrantﬁle deﬁnes many variations of operating systems, releases, and architectures. To see a complete list of the virtual machine options available to you, run the command vagrant status from within the same directory as the Vagrantﬁle. The command may take a few moments to run as the conﬁgurations deﬁned in the Vagrantﬁle are evaluated, and calls are made to the underlying VirtualBox utilities to check for the existence and operational state of each possibility. You should expect to see output along the lines of:
$ vagrant status Current machine states:
 saucy32 not created (virtualbox) raring32 not created (virtualbox) quantal32 not created (virtualbox) precise32 not created (virtualbox)
4.4. Using Vagrant to Test Trafﬁc Server 263 Apache-Trafﬁcserver-Server Documentation, latest
 trusty32 not created (virtualbox) saucy64 not created (virtualbox) raring64 not created (virtualbox) quantal64 not created (virtualbox) precise64 not created (virtualbox) trusty64 running (virtualbox) freebsd not created (virtualbox) omnios not created (virtualbox) lucid64 not created (virtualbox) fedora18 not created (virtualbox) centos63 not created (virtualbox) centos59 not created (virtualbox) centos64 not created (virtualbox) debian7 not created (virtualbox) sles11 not created (virtualbox) oel63 not created (virtualbox)
This environment represents multiple VMs. The VMs are all listed above with their current state. For more information about a specific VM, run `vagrant status NAME`.
Creating and Destroying
Creation and destruction of virtual machines with Vagrant is very simple. To bring a new virtual machine into exis- tence, run the following command from the same directory in which the Vagrantﬁle is located: vagrant up
Where  should be the speciﬁc operating system release you wish to use for the virtual machine. For example, to test Trafﬁc Server in a CentOS 6.4 environment, you would run: vagrant up centos64
Running the vagrant up command for a virtual machine which does not exist yet (or has previously been deleted) will create a brand new virtual machine, using the appropriate image (called a basebox in Vagrant parlance), as well as provision the machine according to any conﬁguration management rules speciﬁed in the Vagrantﬁle. Similarly, you may destroy the virtual machine when you are ﬁnished by running the command: vagrant destroy
Or if you wish to only stop the virtual machine temporarily without deleting it, you may run: vagrant halt
A halted virtual machine is started back up with the same vagrant up command shown earlier. The difference is that Vagrant will recognize the box already exists and do nothing more than start the vm process and conﬁgure the virtual networking interfaces on your host. Any conﬁguration management hooks in the Vagrantﬁle will be skipped.
Logging In
Logging into a virtual machine created with Vagrant may be accomplished in a couple different ways. The easiest is to let Vagrant itself ﬁgure out where the machine is and how to properly authenticate you to it:
264 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 vagrant ssh
Using that command from within the same directory as the Vagrantﬁle allows you to skip ﬁguring out what virtual network interface has been attached to the machine, what local port may be assigned to handle SSH forwarding, and so on. As long as the virtual machine was already running, you will be quickly dropped into a local shell in the virtual machine as the vagrant user.
: Vagrant by default uses a widely-shared private RSA key on newly created virtual machines (that are built on public basebox images). The default user on these baseboxes is also conﬁgured for password-less sudo permissions. This is very clearly insecure, but Vagrant is designed for local testing and development, not production (or other public) uses, so the project has made the philosophical trade-off in favor of ease of use.
Alternatively, you may SSH directly to the virtual machine. Because the virtual machines are conﬁgured to use only the private virtual network layer provided by VirtualBox, you cannot directly. Instead, VirtualBox has created a local port mapping automatically which should be used. There is no ﬁxed, pre-determined port mapping that will be universally valid, as Vagrant and VirtualBox may be used together to run an arbitrary number of virtual machines simultaneously, each provisioned in any order, and deﬁned by any number and variety of Vagrantﬁles. The correct way to determine what port Vagrant and VirtualBox have used to map to a given virtual machine is to run the following command from within the same directory as your Vagrantﬁle: vagrant ssh-config
That will output a conﬁguration block, suitable for inclusion in your local ~/.ssh/config ﬁle. Note speciﬁcally, in addition to the port, the path to the private key you will need to use as your identity when attempting to log into the virtual machine.
Shared Host Folders
VirtualBox provides a facility for mounting directories from your host machine as ﬁlesystems inside the virtual ma- chines. The Trafﬁc Server Vagrantﬁle makes use of this feature to mount its own source tree in a predictable location in the virtual environment. Multiple methods are available for this, including NFS, CIFS, and simulated block devices. The Trafﬁc Server project opts to use NFS for its simplicity, speed, support for features such as symlinks, and wide interoperability across the various guest operating systems included in the Vagrantﬁle. Within the included Vagrantﬁle, you can see the following line: config.vm.synced_folder".","/opt/src/trafficserver.git", :nfs=> true
This directs VirtualBox to mount the directory in which the Trafﬁc Server Vagrantﬁle resides as an NFS mount inside the virtual machine at the path /opt/src/trafficserver.git. Additional host directories may be mounted in the same manner should the need arise.
Forwarding Custom Ports
Building Trafﬁc Server Inside Vagrant
Producing Trafﬁc Server builds from within the Vagrant managed virtual machines is effectively no different than in any other environment. The same directory in which the Vagrantﬁle exists will be mounted via NFS inside the virtual machine at the path /opt/src/trafficserver.git.
4.4. Using Vagrant to Test Trafﬁc Server 265 Apache-Trafﬁcserver-Server Documentation, latest
: If you have run autoconf or configure from outside the virtual machine environment against the same Git working copy as is mounted inside the virtual machine, you will encounter failures should you attempt to run any of the Make targets inside the VM. Any build related commands should be run inside of the virtual machine. Additionally, if you are running more than one virtual machine simultaneously, remember that they are each using the same NFS export on your host machine.
Debugging and Analysis
Debug Builds
A debugger can set breakpoints in a plugin. Use a Trafﬁc Server debug build and compile the plugin with the -g option. A debugger can also be used to analyze a core dump. To generate core, set the size limit of the core ﬁles in the records.config ﬁle to -1 as follows:
CONFIG :ts:cv:`proxy.config.core_limit` INT -1
This is the equivalent of setting ulimit -c unlimited
Debugging Tips:
• Use a Trafﬁc Server debug version. • Use assertions in your plugin (TSAssert() and TSReleaseAssert()).
Using TSAssert
Debug Tags
Use the API void TSDebug (const char *tag, const char *format_str, ...) to add traces in your plugin. In this API: • tag is the Trafﬁc Server parameter that enables Trafﬁc Server to print out ‘‘format_str‘‘ • ... are variables for ‘‘format_str‘‘ in the standard printf style. Run Trafﬁc Server with the -Ttag option. For example, if the tag is my-plugin, then the debug output goes to traffic.out.See below:
 traffic_server-T"my-plugin"
Set the following variables in records.config (in the Trafﬁc Server config directory):
CONFIG proxy.config.diags.debug.enabled INT1 CONFIG proxy.config.diags.debug.tags STRING debug-tag-name
In this case, debug output goes to traffic.out. Example:
TSDebug ("my-plugin","Starting my-plugin at %d \n", the_time);
266 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
The statement "Starting my-plugin at " appears whenever you run Trafﬁc Server with the my-plugin tag:
 traffic_server-T"my-plugin"
Other Useful Internal Debug Tags
Embedded in the base Trafﬁc Server code are many debug tags for internal debugging purposes. These can also be used to follow Trafﬁc Server behavior for testing and analysis. The debug tag setting (-T and proxy.config.diags.debug.tags) is a anchored regular expression against which the tag for a speciﬁc debug message is matched. This means the value “http” matches debug emssages with the tag “http” but also “http_seq” and “http_trans”. If you want multiple tags then use a pipe symbol to separate the tags. For example “http_tproxy|dns|hostdb” will match any of the message tags “http_tproxy”, “dns”, “hostdb”, or “dns_srv” (but not “http_trans” nor “splitdns”). Some of the useful HTTP debug tags are: • http_hdrs - traces all incoming and outgoing HTTP headers. • http_trans - traces actions in an HTTP transaction. • http_seq - traces the sequence of the HTTP state machine. • http_tproxy - transparency related HTTP events • dns - DNS operations • hostdb - Host name lookup • iocore_net - Socket and low level IO (very voluminous) • socket - socket operations • ssl - SSL related events • cache - Cache operations (many subtags, examine the output to narrow the tag set) • cache_update - Cache updates including writes • cache_read - Cache read events. • dir_probe - Cache searches. • sdk - gives some warning concerning API usage.
Core Dump Analysis
Proﬁling
Memory Leaks
Memory leaks in a plugin can be detected using e.g. an MRTG graph related to memory - you can use memory dump information. Enable mem dump in records.config as follows:
CONFIG proxy.config.dump_mem_info_frequency INT
This causes Trafﬁc Server to dump memory information to traffic.out at  (intervals are in seconds). A zero value means that it is disabled.
4.5. Debugging and Analysis 267 Apache-Trafﬁcserver-Server Documentation, latest
Cache Architecture
The original architectural documents for Trafﬁc Server were lost in the transition to an open source project. The documents in this section are provisional and were written based on the existing code. The purpose is to have a high level description of aspects of Trafﬁc Server to better inform ongoing work. In the ﬁnal section on “hacking” we try to document our approaches to understanding and modifying the source.
Cache Architecture
Introduction
In addition to being an HTTP proxy, Apache Trafﬁc Server™ is also an HTTP cache. Trafﬁc Server can cache any octet stream, although it currently supports only those octet streams delivered by the HTTP protocol. When such a stream is cached (along with the HTTP protocol headers) it is termed an object in the cache. Each object is identiﬁed by a globally unique value called a cache key. The purpose of this document is to describe the basic structure and implementation details of the Trafﬁc Server cache. Conﬁguration of the cache will be discussed only to the extent needed to understand the internal mechanisms. This document will be useful primarily to Trafﬁc Server developers working on the Trafﬁc Server codebase or plugins for Trafﬁc Server. It is assumed the reader is already familiar with the Administrator’s Guide and speciﬁcally with HTTP Proxy Caching and along with the associated conﬁguration ﬁles. Unfortunately, the internal terminology is not particularly consistent, so this document will frequently use terms in different ways than they are used in the code in an attempt to create some consistency.
Cache Layout
The following sections describe how persistent cache data is structured. Trafﬁc Server treats its persisent storage as an undifferentiated collection of bytes, assuming no other structure to it. In particular, it does not use the ﬁle system of the host operating system. If a ﬁle is used it is used only to mark out the set of bytes to be used.
Cache storage
The raw storage for the Trafﬁc Server cache is conﬁgured in storage.config. Each line in the ﬁle deﬁnes a cache span which is treated as a uniform persistent store.
4.1: Two cache spans
This storage organized into a set of cache volumes which are deﬁned in volume.config. These are the units that are used for all other administator level conﬁguration. Cache volumes can be deﬁned by a percentage of the total storage or as an absolute amount of storage. By default, each cache volume is spread across all of the cache spans for robustness. The intersection of a cache volume and a cache span is a cache stripe. Each cache span is divided into cache stripes and each cache volume is a collection of those stripes.
268 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
If the cache volumes for the example cache spans were deﬁned as:
Then the actual layout would look like:
Cache stripes are the fundamental unit of cache for the implementation. A cached object is stored entirely in a single stripe, and therefore in a single cache span. Objects are never split across cache spans or volumes. Objects are assigned to a stripe (and in turn to a cache volume) automatically based on a hash of the URI used to retrieve the object from the origin server. It is possible to conﬁgure this to a limited extent in hosting.config, which supports content from speciﬁc hosts or domain to be stored on speciﬁc cache volumes. As of version 4.0.1 it is also possible to control which cache spans (and hence, which cache stripes) are contained in a speciﬁc cache volume. The layout and structure of the cache spans, the cache volumes, and the cache stripes that compose them are derived en- tirely from storage.config and cache.config and is recomputed from scratch when the traffic_server is started. Therefore, any change to those ﬁles can (and almost always will) invalidate the existing cache in its entirety.
Stripe Structure
Trafﬁc Server treats the storage associated with a cache stripe as an undifferentiated span of bytes. Internally each stripe is treated almost entirely independently. The data structures described in this section are duplicated for each stripe. Internally the term volume is used for these stripes and implemented primarily in Vol. What a user thinks of as a volume (and what this document calls a cache volume) is represented by CacheVol.
: Stripe assignment must be done before working with an object because the directory is local to the stripe. Any cached objects for which the stripe assignment is changed are effectively lost as their directory data will not be found in the new stripe.
Cache Directory
Content in a stripe is tracked via a directory. Each element of the directory is a directory entry and is represented by Dir. Each entry refers to a chunk of contiguous storage in the cache. These are referred to variously as fragments, segments, docs, documents, and a few other things. This document will use the term fragment as that is the most common reference in the code. The term Doc (for Doc) will be used to refer to the header data for a fragment. Overall, the directory is treated as a hash with the cache ID as the key. See directory probing for how the cache ID is used to locate a directory entry. The cache ID is in turn computed from a cache key which by default is the URL of the content. The directory is used as a memory resident structure, which means a directory entry is as small as possible (currently 10 bytes). This forces some compromises on the data that can be stored there. On the other hand this means that most cache misses do not require disk I/O, which has a large performance beneﬁt.
4.6. Cache Architecture 269 Apache-Trafﬁcserver-Server Documentation, latest
The directory is always fully sized; once a stripe is initialized the directory size is ﬁxed and never changed. This size is related (roughly linearly) to the size of the stripe. It is for this reason the memory footprint of Trafﬁc Server depends strongly on the size of the disk cache. Because the directory size does not change, neither does this memory requirement, so Trafﬁc Server does not consume more memory as more content is stored in the cache. If there is enough memory to run Trafﬁc Server with an empty cache there is enough to run it with a full cache.
Each entry stores an offset in the stripe and a size. The size stored in the directory entry is an approximate size which is at least as big as the actual data in the fragment. Exact size data is stored in the fragment header on disk.
: Data in HTTP headers cannot be examined without disk I/O. This includes the original URL for the object. The cache key is not stored explicitly and therefore cannot be reliably retrieved.
The directory is a hash table that uses chaining for collision resolution. Because each entry is small they are used directly as the list header of the hash bucket. Chaining is implemented by imposing grouping structures on the entries in a directory. The ﬁrst level grouping is a directory bucket. This is a ﬁxed number (currently 4, deﬁned as DIR_DEPTH) of entries. This serves to deﬁne the basic hash buckets with the ﬁrst entry in each cache bucket serving as the root of the hash bucket.
: The term bucket is used in the code to mean both the conceptual bucket for hashing and for a structural grouping mechanism in the directory. These will be qualiﬁed as needed to distinguish them. The unqualiﬁed term bucket is almost always used to mean the structural grouping in the directory.
Directory buckets are grouped in to segments. All segments in a stripe have the same number of buckets. The number of segments in a stripe is chosen so that each segment has as many buckets as possible without exceeding 65,535 (216-1) entries in a segment.
Each directory entry has a previous and next index value which is used to link entries in the same segment. Because no segment has more than 65,535 entries, 16 bits sufﬁces for storing the index values. The stripe header contains an array of entry indices which are used as the roots of entry free lists, one for each segment. Active entries are stored via the bucket structure. When a stripe is initialized the ﬁrst entry in each bucket is zeroed (marked unused) and all other entries are put in the corresponding segment free list in the stripe header. This means the ﬁrst entry of each directory bucket is used as the root of a hash bucket and is therefore marked unused rather than being put a free list. The other
270 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest entries in the directory bucket are preferred for adding to the corresponding hash bucket but this is not required. The segment free lists are initialized such that the extra bucket entries are added in order; all the seconds, then the thirds, then the fourths. Because the free lists are FIFOs, this means extra entries will be selected from the fourth entries across all the buckets ﬁrst, then the thirds, etc. When allocating a new directory entry in a bucket the entries are searched from ﬁrst to last, which maximizes bucket locality (that is, cache IDs that map to the same hash bucket will also tend to use the same directory bucket).
Entries are removed from the free list when used and returned when no longer in use. When a fragment needs to be put in to the directory the cache ID is used to locate a hash bucket (which also determines the segment and directory bucket). If the ﬁrst entry in the directory bucket is marked unused, it is used. Otherwise, the other entries in the bucket are searched and if any are on the free list, that entry is used. If none are available then the ﬁrst entry on the segment free list is used. This entry is attached to the hash bucket via the same next and previous indices used for the free list so that it can be found when doing a lookup of a cache ID.
Storage Layout
The storage layout is the stripe metadata followed by cached content. The metadata consists of three parts: the stripe header, the directory, and the stripe footer. The metadata is stored twice. The header and the footer are instances of VolHeaderFooter. This is a stub structure which can have a trailing variable sized array. This array is used as the segment free list roots in the directory. Each contains the segment index of the ﬁrst element of the free list for the segment. The footer is a copy of the header without the segment free lists. This makes the size of the header dependent on the directory but not that of the footer.
Each stripe has several values that describe its basic layout: skip The start of stripe data. This represents either space reserved at the start of a physical device to avoid problems with the host operating system, or an offset representing use of space in the cache span by other stripes. start The offset for the start of the content, after the stripe metadata. length Total number of bytes in the stripe. Vol::len.
4.6. Cache Architecture 271 Apache-Trafﬁcserver-Server Documentation, latest
 data length Total number of blocks in the stripe available for content storage. Vol::data_blocks.
: Great care must be taken with sizes and lengths in the cache code because there are at least three different metrics (bytes, cache blocks, store blocks) used in various places.
The total size of the directory (the number of entries) is computed by taking the size of the cache stripe and dividing by the average object size. The directory always consumes this amount of memory which has the effect that if cache size is increased so is the memory requirement for Trafﬁc Server. The average object size defaults to 8000 bytes but can be conﬁgured using :ts:cv:‘proxy.conﬁg.cache.min_average_object_size‘. Increasing the average object size will reduce the memory footprint of the directory at the expense of reducing the number of distinct objects that can be stored in the cache. The content area stores the actual objects and is used as a circular buffer where new objects overwrite the least recently cached objects. The location in a stripe where new cache data is written is called the write cursor. This means that objects can be de facto evicted from cache even if they have not expired if the data is overwritten by the write cursor. If an object is overwritten this is not detected at that time and the directory is not updated. Instead it will be noted if the object is accessed in the future and the disk read of the fragment fails.
4.2: The write cursor and documents in the cache.
: Cache data on disk is never updated.
This is a key thing to keep in mind. What appear to be updates (such as doing a refresh on stale content and getting back a 304) are actually new copies of data being written at the write cursor. The originals are left as “dead” space which will be consumed when the write cursor arrives at that disk location. Once the stripe directory is updated (in memory) the original fragment in the cache is effectively destroyed. This is the general space management technique used in other cases as well. If an object needs to removed from cache, only the directory needs to be changed. No other work (and particularly, no disk I/O) needs to be done.
Object Structure
Objects are stored as two types of data: metadata and content data. Metadata is all the data about the object and the content and includes the HTTP headers. The content data is the content of the object, the octet stream delivered to the client as the object. Objects are rooted in a Doc structure stored in the cache. Doc serves as the header data for a cache fragment and is contained at the start of every fragment. The ﬁrst fragment for an object is termed the ﬁrst Doc and always contains the object metadata. Any operation on the object will read this fragment ﬁrst. The fragment is located by converting the cache key for the object to a cache ID and then doing a lookup for a directory entry with that key. The directory entry has the offset and approximate size of the ﬁrst fragment which is then read from the disk. This fragment will contain the request header and response along with overall object properties (such as content length).
272 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Trafﬁc Server supports varying content for objects. These are called alternates. All metadata for all alternates is stored in the ﬁrst fragment including the set of alternates and the HTTP headers for them. This enables alternate selection to be done after the ﬁrst Doc is read from disk. An object that has more than one alternate will have the alternate content stored separately from the ﬁrst fragment. For objects with only one alternate the content may or may not be in the same (ﬁrst) fragment as the metadata. Each separate alternate content is allocated a directory entry and the key for that entry is stored in the ﬁrst fragment metadata. Prior to version 4.0.1, the header data was stored in the CacheHTTPInfoVector class which was marshaled to a variable length area of the on disk image, followed by information about additional fragments if needed to store the object.
4.3: Doc layout 3.2.0
This had the problem that with only one fragment table it could not be reliable for objects with more than one alternate[#multiple-alternates]_. Therefore, the fragment data was moved from being a separate variable length section of the metadata to being directly incorporated in to the CacheHTTPInfoVector, yielding a layout of the following form.
4.4: Doc layout 4.0.1
Each element in the vector contains for each alternate, in addition to the HTTP headers and the fragment table (if any), a cache key. This cache key identiﬁes a directory entry that is referred to as the earliest Doc. This is the location where the content for the alternate begins. When the object is ﬁrst cached, it will have a single alternate and that will be stored (if not too large) in ﬁrst Doc. This is termed a resident alternate in the code. This can only happen on the initial store of the object. If the metadata is updated (such as a 304 response to an If-Modified-Since request) then unless the object is small, the object data will be left in the original fragment and a new fragment written as the ﬁrst fragment, making the alternate non-resident. Small is deﬁned as a length smaller than :ts:cv:‘proxy.conﬁg.cache.alt_rewrite_max_size‘.
: The CacheHTTPInfoVector is stored only in the ﬁrst Doc. Subsequent Doc instances for the object, including the earliest Doc, should have an hlen of zero and if not, it is ignored.
Large objects are split in to multiple fragments when written to the cache. This is indicated by a total document length that is longer than the content in ﬁrst Doc or an earliest Doc. In such a case a fragment offset table is stored. This contains the byte offset in the object content of the ﬁrst byte of content data for each fragment past the ﬁrst (as the offset for the ﬁrst is always zero). This allows range requests to be serviced much more efﬁciently for large objects, as intermediate fragments that do not contain data in the range can be skipped. The last fragment in the sequence is detected by the fragment size and offset reaching the end of the total size of the object, there is no explicit end mark. Each fragment is computationally chained from the previous in that the cache key for fragment N is computed by:
4.6. Cache Architecture 273 Apache-Trafﬁcserver-Server Documentation, latest
 key_for_N_plus_one= next_key(key_for_N);
Where next_key is a global function that deterministically computes a new cache key from an existing cache key. Objects with multiple fragments are laid out such that the data fragments (including the earliest Doc) are written ﬁrst and the ﬁrst Doc is written last. When read from disk, both the ﬁrst and earliest Doc are validated (tested to ensure that they haven’t been overwritten by the write cursor) to verify that the entire document is present on disk (as they bookend the other fragments - the write cursor cannot overwrite them without overwriting at least one of the veriﬁed Doc instances). Note that while the fragments of a single object are ordered they are not necessarily contiguous, as data from different objects are interleaved as the data arrives in Trafﬁc Server.
4.5: Multi-alternate and multi-fragment object storage
Documents which are pinned into the cache must not be overwritten so they are evacuated from in front of the write cursor. Each fragment is read and rewritten. There is a special lookup mechanism for objects that are being evacuated so that they can be found in memory rather than the potentially unreliable disk regions. The cache scans ahead of the write cursor to discover pinned objects as there is a dead zone immediately before the write cursor from which data cannot be evacuated. Evacuated data is read from disk and placed in the write queue and written as its turn comes up. Objects can only be pinned via cache.config and while :ts:cv:‘proxy.conﬁg.cache.permit.pinning‘ is set to non- zero (it is zero by default). Objects which are in use when the write cursor is near use the same underlying evacuation mechanism but are handled automatically and not via the explicit pinned bit in Dir.
Additional Notes
Some general observations on the data structures.
Cyclone buffer
Because the cache is a cyclone cache, objects are not preserved for an indeﬁnite time. Even if the object is not stale it can be overwritten as the cache cycles through its volume. Marking an object as pinned preserves the object through the passage of the write cursor but this is done by copying the object across the gap, in effect re-storing it in the cache. Pinning large objects or a large number objects can lead to excessive disk activity. The original purpose of pinning was for small, frequently used objects explicitly marked by the administrator. This means the purpose of expiration data on objects is simply to prevent them from being served to clients. They are not in the standard sense deleted or cleaned up. The space can’t be immediately reclaimed in any event, because writing only happens at the write cursor. Deleting an object consists only of removing the directory entries in the volume directory which sufﬁces to (eventually) free the space and render the document inaccessible.
274 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Historically, the cache was designed this way because web content was relatively small and not particularly consistent. The design also provided high performance and low consistency requirements. There are no fragmentation issues for the storage, and both cache misses and object deletions require no disk I/O. It does not deal particularly well with long term storage of large objects. See the Volume Tagging appendix for details on some work in this area.
Disk Failure
The cache is designed to be relatively resistant to disk failures. Because each storage unit in each cache volume is mostly independent, the loss of a disk simply means that the corresponding Vol instances (one per cache volume that uses the storage unit) becomes unusable. The primary issue is updating the volume assignment table to both preserve assignments for objects on still operational volumes while distributing the assignments from the failed disk to those operational volumes. This mostly done in:
AIO_Callback_handler::handle_disk_failure
Restoring a disk to active duty is a more difﬁcult task. Changing the volume assignment of a cache key renders any currently cached data inaccessible. This is not a problem when a disk has failed, but is a bit trickier to decide which cached objects are to be de facto evicted if a new storage unit is added to a running system. The mechanism for this, if any, is still under investigation.
Implementation Details
Stripe Directory
The in memory volume directory entries are described below. class Dir Deﬁned in P_CacheDir.h. Name Type Use offset unsigned int:24 Offset of ﬁrst byte of metadata (volume relative) big unsigned in:2 Size multiplier size unsigned int:6 Size tag unsigned int:12 Partial key (fast collision check) phase unsigned int:1 Phase of the Doc (for dir valid check) head unsigned int:1 Flag: ﬁrst fragment in an object pinned unsigned int:1 Flag: document is pinned token unsigned int:1 Flag: Unknown next unsigned int:16 Segment local index of next entry. offset_high inku16 High order offset bits The stripe directory is an array of Dir instances. Each entry refers to a span in the volume which contains a cached object. Because every object in the cache has at least one directory entry this data has been made as small as possible. The offset value is the starting byte of the object in the volume. It is 40 bits long, split between the offset (lower 24 bits) and offset_high (upper 16 bits) members. Note that since there is a directory for every storage unit in a cache volume, this is the offset in to the slice of a storage unit attached to that volume. The size and big values are used to calculate the approximate size of the fragment which contains the object. This value is used as the number of bytes to read from storage at the offset value. The exact size is contained in the object metadata in Doc which is consulted once the read has completed. For this reason, the approximate size needs to be at least as large as the actual size but can be larger, at the cost of reading the extraneous bytes. The computation of the approximate size of the fragment is deﬁned as:
4.6. Cache Architecture 275 Apache-Trafﬁcserver-Server Documentation, latest
( *size* +1) * 2^ ( CACHE_BLOCK_SHIFT+3 ** big* )
Where CACHE_BLOCK_SHIFT is the bit width of the size of a basic cache block (9, corresponding to a sector size of 512). Therefore the value with current deﬁnes is:
( *size* +1) * 2^(9+3 ** big*)
Because big is 2 bits, the values for the multiplier of size are: big Multiplier Maximum Size 0 512 (2^9) 32768 (2^15) 1 4096 (2^12) 262144 (2^18) 2 32768 (2^15) 2097152 (2^21) 3 262144 (2^18) 16777216 (2^24) Note also that size is effectively offset by one, so a value of 0 indicates a single unit of the multiplier. The target fragment size can set with the records.config value :ts:cv:‘proxy.conﬁg.cache.target_fragment_size‘. This value should be chosen so that it is a multiple of a cache entry multiplier. It is not necessary to make it a power of two[#cache-mult-value]_. Larger fragments increase I/O efﬁciency but lead to more wasted space. The default size (1M, 2^20) is a reasonable choice in most circumstances, altough in very speciﬁc cases there can be beneﬁt from tuning this parameter. Trafﬁc Server imposes an internal maximum of a 4,194,232 bytes, which is 4M (2^22), less the size of a struct Doc. In practice, the largest reasonable target fragment size is 4M - 262,144 = 3,932,160. When a fragment is stored to disk, the size data in the cache index entry is set to the ﬁnest granularity permitted by the size of the fragment. To determine this, consult the cache entry multipler table and ﬁnd the smallest maximum size that is at least as large as the fragment. That will indicate the value of big selected and therefore the granularity of the approximate size. That represents the largest possible amount of wasted disk I/O when the fragment is read from disk. The set of index entries for a volume are grouped in to segments. The number of segments for an index is selected so that there are as few segments as possible such that no segment has more than 2^16 entries. Intra-segment references can therefore use a 16 bit value to refer to any other entry in the segment. Index entries in a segment are grouped buckets, each of DIR_DEPTH (currently 4) entries. These are handled in the standard hash table manner, giving somewhat less than 2^14 buckets per segment.
Directory Probing
Directory probing is the locating of a speciﬁc directory entry in the stripe directory based on a cache ID. This is handled primarily by the function dir_probe(). This is passed the cache ID (:arg:‘key‘), a stripe (:arg:‘d‘), and a last collision (:arg:‘last_collision‘). The last of these is an in and out parameter, updated as useful during the probe. Given an ID, the top half (64 bits) is used as a segment index, taken modulo the number of segments in the directory. The bottom half is used as a bucket index, taken modulo the number of buckets per segment. The :arg:‘last_collision‘ value is used to mark the last matching entry returned by dir_probe(). After computing the appropriate bucket, the entries in that bucket are searched to ﬁnd a match. In this case a match is detected by comparison of the bottom 12 bits of the cache ID (the cache tag). The search starts at the base entry for the bucket and then proceeds via the linked list of entries from that ﬁrst entry. If a tag match is found and there is no :arg:‘collision‘ then that entry is returned and :arg:‘last_collision‘ is updated to that entry. If :arg:‘collision‘ is set and if it isn’t the current match, the search continues down the linked list, otherwise :arg:‘collision‘ is cleared and the search continues. The effect of this is that matches are skipped until the last returned match (:arg:‘last_collision‘) is found, after which the next match (if any) is returned. If the search falls off the end of the linked list, then a miss result is returned (if no last collision), otherwise the probe is restarted after clearing the collision on the presumption that the entry for the
276 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 collision has been removed from the bucket. This can lead to repeats among the returned values but guarantees that no valid entry will be skipped. Last collision can therefore be used to restart a probe at a later time. This is important because the match returned may not be the actual object. Although the hashing of the cache ID to a bucket and the tag matching is unlikely to create false positives, it is possible. When a fragment is read the full cache ID is available and checked and if wrong, that read can be discarded and the next possible match from the directory found because the cache virtual connection tracks the last collision value.
Cache Operations
Cache activity starts after the HTTP request header has been parsed and remapped. Tunneled transactions do not interact with the cache because the headers are never parsed. To understand the logic we must introduce the term cache valid which means something that is directly related to an object that is valid to be put in the cache (e.g. a DELETE which refers to a URL that is cache valid but cannot be cached itself). This is important because Trafﬁc Server computes cache validity several times during a transaction and only performs cache operations for cache valid results. The criteria used changes during the course of the transaction as well. This is done to avoid the cost of cache activity for objects that cannot be in the cache. The three basic cache operations are: lookup, read, and write. We will take deleting entries as a special case of writing where only the volume directory is updated. After the client request header is parsed and is determined to be potentially cacheable, a cache lookup is done. If successful, a cache read is attempted. If either the lookup or the read fails and the content is considered cacheable then a cache write is attempted.
Cacheability
The ﬁrst thing done with a request with respect to cache is to determine whether it is potentially a valid object for the cache. After initial parsing and remapping, this check is done primarily to detect a negative result, as it allows further cache processing to be skipped. It will not be put in to the cache, nor will a cache lookup be performed. There are a number of prerequisites along with conﬁguration options to change them. Additional cacheability checks are done later in the process, when more is known about the transaction (such as plugin operations and the origin server response). Those checks are described as appropriate in the sections on the relevant operations. The set of things which can affect cacheability are: • Built in constraints. • Settings in records.config. • Settings in cache.config. • Plugin operations. The initial internal checks, along with their records.config overrides[#cacheability-overrides]_, are done in HttpTransact::is_request_cache_lookupable. The checks that are done are: Cacheable Method The request must be one of GET, HEAD, POST, DELETE, PUT. See HttpTransact::is_method_cache_lookupable(). Dynamic URL Trafﬁc Server tries to avoid caching dynamic content because it’s dynamic. A URL is considered dynamic if: • It is not HTTP or HTTPS,
4.6. Cache Architecture 277 Apache-Trafﬁcserver-Server Documentation, latest
• Has query parameters, • Ends in asp, • Has cgi in the path. This check can be disabled by setting a non-zero value for :ts:cv:‘proxy.conﬁg.http.cache.cache_urls_that_look_dynamic‘. In addition if a TTL is set for rule that matches in cache.config then this check is not done. Range Request Cache valid only if :ts:cv:‘proxy.conﬁg.http.cache.range.lookup‘ in records. config is non-zero. This does not mean the range request can be cached, only that it might be satisﬁable from the cache. In addition, :ts:cv:‘proxy.conﬁg.http.cache.range.write‘ can be set to try to force a write on a range request. This probably has little value at the moment, but if for ex- ample the origin server ignores the Range: header, this option can allow for the response to be cached. It is disabled by default, for best performance. A plugin can call TSHttpTxnReqCacheableSet() to force the request to be viewed as cache valid.
Cache Lookup
If the initial request is not determined to be cache invalid then a lookup is done. Cache lookup determines if an object is in the cache and if so, where it is located. In some cases the lookup proceeds to read the ﬁrst Doc from disk to verify the object is still present in the cache. The basic steps to a cache lookup are: 1. The cache key is computed. This is normally computed using the request URL but it can be overridden by a plugin . The cache index string is not stored, as it is presumed computable from the client request headers. 2. The cache stripe is determined (based on the cache key). The cache key is used as a hash key in to an array of Vol instances. The construction and arrangement of this array is the essence of how volumes are assigned. 3. The cache stripe directory is probed using the index key computed from the cache key. Various other lookaside directories are checked as well, such as the aggregation buffer. 4. If the directory entry is found the ﬁrst Doc is read from disk and checked for validity. This is done in CacheVC::openReadStartHead() or CacheVC::openReadStartEarliest() which are tightly coupled methods. If the lookup succeeds, then a more detailed directory entry (struct OpenDir) is created. Note that the directory probe includes a check for an already extant OpenDir which, if found, is returned without additional work.
Cache Read
Cache read starts after a successful cache lookup. At this point the ﬁrst Doc has been loaded in to memory and can be consulted for additional information. This will always contain the HTTP headers for all alternates of the object.
Read while write
There is provision in the code to support read while write, that is, serving an object from cache in one transaction while it is being written in another. Several settings are needed for it to be used. See Reducing Origin Server
278 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Requests (Avoiding the Thundering Herd). It must speciﬁcally enabled in records.config and if not, a cache read will fail if the object is currently be written or updated.
At this point an alternate for the object is selected. This is done by comparing the client request to the stored response headers, but it can be controlled by a plugin using TS_HTTP_ALT_SELECT_HOOK. The content can now be checked to see if it is stale by calculating the freshness of the object. This is essentially checking how old the object is by looking at the headers and possibly other metadata (note that the headers can’t be checked until we’ve selected an alternate). Most of this work is done in HttpTransact::what_is_document_freshness. First, the TTL (time to live) value, which can be set in cache.config, is checked if the request matches the conﬁguration ﬁle line. This is done based on when the object was placed in the cache, not on any data in the headers. Next, an internal ﬂag (needs-revalidate-once) is checked if the cache.config value revalidate-after is not set, and if set the object is marked stale. After these checks the object age is calculated by HttpTransactHeaders::calculate_document_age. and then any conﬁgured fuzzing is applied. The limits to this age based on available data is calculated by HttpTransact::calculate_document_freshness_limit. How this age is used is determined by the records.config setting for :ts:cv:‘proxy.conﬁg.http.cache.when_to_revalidate‘. If this is 0 then the built caclulations are used which compare the freshness limits with document age, modiﬁed by any of the client supplied cache control values (max-age, min-fresh, max-stale) unless explicitly overridden in cache.config. If the object is not stale then it is served to the client. If it is stale, the client request may be changed to an If Modified Since request to revalidate. The request is served using a standard virtual connection tunnel (HttpTunnel) with the CacheVC acting as the producer and the client NetVC acting as the sink. If the request is a range request this can be modiﬁed with a transform to select the appropriate parts of the object or, if the request contains a single range, it can use the range acceleration. Range acceleration is done by consulting a fragment offset table attached to the earliest Doc which contains offsets for all fragments past the ﬁrst. This allows loading the fragment containing the ﬁrst requested byte immediately rather than performing reads on the intermediate fragments.
Cache Write
Writing to the cache is handled by an instance of the class CacheVC. This is a virtual connection which receives data and writes it to cache, acting as a sink. For a standard transaction data transfers between virtual connections (VConns) are handled by HttpTunnel. Writing to the cache is done by attaching a CacheVC instance as a tunnel consumer. It therefore operates in parallel with the virtual connection that transfers data to the client. The data does not ﬂow to the cache and then to the client, it is split and goes both directions in parallel. This avoids any data synchronization issues between the two.
Writing to disk
The actual write to disk is handled in a separate thread dedicated to I/O operations, the AIO threads. The cache logic marshals the data and then hands the operation off to the AIO thread which signals back once the operation completes.
While each CacheVC handles its transactions independently, they do interact at the volume level as each CacheVC makes calls to the volume object to write its data to the volume content. The CacheVC accumulates data internally
4.6. Cache Architecture 279 Apache-Trafﬁcserver-Server Documentation, latest until either the transaction is complete or the amount of data to write exceeds the target fragment size. In the former case the entire object is submitted to the volume to be written. In the latter case, a target fragment size amount of data is submitted and the CacheVC continues to operate on subsequent data. The volume in turn places these write requests in an holding area called the aggregation buffer. For objects under the target fragment size, there is no consideration of order, the object is simply written to the volume content. For larger objects, the earliest Doc is written ﬁrst and the ﬁrst Doc written last. This provides some detection ability should the object be overwritten. Because of the nature of the write cursor no fragment after the ﬁrst fragment (in the earliest Doc) can be overwritten without also overwriting that ﬁrst fragment (since we know at the time the object was ﬁnalized in the cache the write cursor was at the position of the ﬁrst Doc).
: It is the responsibility of the CacheVC to not submit writes that exceed the target fragment size.
Update
Cache write also covers the case where an existing object in the cache is modiﬁed. This occurs when: • A conditional request is made to the origin server and a 304 - Not Modified response is received. • An alternate of the object is retrieved from an origin server and added to the object. • An alternate of the object is removed (e.g., due to a DELETE request). In every case the metadata for the object must be modiﬁed. Because Trafﬁc Server never updates data already in the cache this means the ﬁrst Doc will be written to the cache again and the volume directory entry updated. Because a client request has already been processed the ﬁrst Doc has been read from cache and is in memory. The alternate vector is updated as appropriate (an entry added or removed, or changed to contain the new HTTP headers), and then written to disk. It is possible for multiple alternates to be updated by different CacheVC instances at the same time. The only contention is the ﬁrst Doc; the rest of the data for each alternate is completely independent.
Aggregation Buffer
Disk writes to cache are handled through an aggregation buffer. There is one for each Vol instance. To minimize the number of system calls data is written to disk in units of roughly target fragment size bytes. The algorithm used is simple: data is piled up in the aggregation buffer until no more will ﬁt without going over the target fragment size, at which point the buffer is written to disk and the volume directory entries for objects with data in the buffer are updated with the actual disk locations for those objects (which are determined by the write to disk action). After the buffer is written it is cleared and process repeats. There is a special lookup table for the aggregation buffer so that object lookup can ﬁnd cache data in that memory. Because data in the aggregation buffer is visible to other parts of the cache, particularly cache lookup, there is no need to push a partially ﬁlled aggregation buffer to disk. In effect, any such data is memory cached until enough additional cache content arrives to ﬁll the buffer. The target fragment size has little effect on small objects because the fragment size is used only to parcel out disk write operations. For larger objects the effect very signiﬁcant as it causes those objects to be broken up in to fragments at different locations on in the volume. Each fragment write has its own entry in the volume directory which are com- putationally chained (each cache key is computed from the previous one). If possible, a fragment table is accumulated in the earliest Doc which has the offsets of the ﬁrst byte for each fragment.
280 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Evacuation Mechanics
By default, the write cursor will overwrite (de facto evict from cache) objects as it proceeds once it has gone around the cache stripe at least once. In some cases this is not acceptable and the object is evacuated by reading it from the cache and then writing it back to cache which moves the physical storage of the object from in front of the write cursor to behind the write cursor. Objects that are evacuated are handled in this way based on data in stripe data structures (attached to the Vol instance). Evacuation data structures are deﬁned by dividing up the volume content into a disjoint and contiguous set of regions of EVACUATION_BUCKET_SIZE bytes. The Vol::evacuate member is an array with an element for each evacuation region. Each element is a doubly linked list of EvacuationBlock instances. Each instance contains a Dir that speciﬁes the fragment to evacuate. It is assumed that an evacuation block is placed in the evacuation bucket (array element) that corresponds to the evacuation region in which the fragment is located although no ordering per bucket is enforced in the linked list (this sorting is handled during evacuation). Objects are evacuated by specifying the ﬁrst or earliest fragment in the evactuation block. The evactuation operation will then continue the evacuation for subsequent fragments in the object by adding those fragments in evacuation blocks. Note that the actual evacuation of those fragments is delayed until the write cursor reaches the fragments, it is not necessarily done at the time the earliest fragment is evacuated. There are two types of evacuations: reader based and forced. The EvacuationBlock has a reader count to track this. If the reader count is zero, then it is a forced evacuation and the the target, if it exists, will be evacuated when the write cursor gets close. If the reader value is non-zero then it is a count of entities that are currently expecting to be able to read the object. Readers increment the count when they require read access to the object, or create the EvacuationBlock with a count of 1. When a reader is ﬁnished with the object it decrements the count and removes the EvacuationBlock if the count goes to zero. If the EvacuationBlock already exists with a count of zero, the count is not modiﬁed and the number of readers is not tracked, so the evacuation is valid as long as the object exists. Evacuation is driven by cache writes, essentially in Vol::aggWrite. This method processes the pending cache virtual connections that are trying to write to the stripe. Some of these may be evacuation virtual connections. If so then the completion callback for that virtual connection is called as the data is put in to the aggregation buffer. When no more cache virtual connections can be processed (due to an empty queue or the aggregation buffer ﬁlling) then Vol::evac_range is called to clear the range to be overwritten plus an additional EVACUATION_SIZE range. The buckets covering that range are checked. If there are any items in the buckets a new cache virtual connection (a doc evacuator) is created and used to read the evacuation item closest to the write cursor (i.e. with the smallest offset in the stripe) instead of the aggregation write proceeding. When the read completes it is checked for validity and if valid, the cache virtual connection for it is placed at the front of the write queue for the stripe and the write aggregation resumed. Before doing a write, the method Vol::evac_range() is called to start an evacuation. If any fragments are found in the buckets in the range the earliest such fragment (smallest offset, closest to the write cursor) is selected and read from disk and the aggregation buffer write is suspended. The read is done via a cache virtual connection which also effectively serves as the read buffer. Once the read is complete, that cache virtual connection instance (the doc evacuator) is placed at the front of the stripe write queue and written out in turn. Because the fragment data is now in memory it is acceptable to overwrite the disk image. Note that when normal stripe writing is resumed, this same check is done again, each time evauating (if needed) a fragment and queuing them for writing in turn. Updates to the directory are done when the write for the evacuated fragment completes. Multi-fragment objects are detected after the read completes for a fragment. If it is not the ﬁrst fragment then the next fragment is marked for evacuation (which in turn, when it is read, will pull the subsequent fragment). The logic presumes that the end of the alternate is when the next key is not in the directory. This interacts with the one at a time strategy of the aggregation write logic. If a fragment is close to the fragment being evacuated, it may end up in the same evacuation bucket. Because the aggregation write checks every time for the next fragment to evacuate it will ﬁnd that next fragment and evacuate it before it is overwritten.
4.6. Cache Architecture 281 Apache-Trafﬁcserver-Server Documentation, latest
Evacuation Operation
The primary source of fragments to be evacuated are active fragments. That is, fragments which are currently open for reading or writing. This is tracked by the reader value in the evacuation blocks noted above. If object pinning is enabled, then a scan is done on a regular basis as the write cursor moves to detect pinned objects and mark them for evacuation. Fragments can also be evacuated through hit evacuation. This is conﬁgured by :ts:cv:‘proxy.conﬁg.cache.hit_evacuate_percent‘ and :ts:cv:‘proxy.conﬁg.cache.hit_evacuate_size_limit‘. When a fragment is read it is checked to see if it is close and in front of the write cursor, close being less than the speciﬁed percent of the size of the stripe. If set at the default value of 10, then if the fragment is withing 10% of the size of the stripe, it is marked for evacuation. This is cleared if the write cursor passes through the fragment while it remains open (as all open objects are evacuated). If, when the object is closed, the fragment is still marked then it is placed in the appropriate evacuation bucket.
Initialization
Initialization starts with an instance of Store reading the storage conﬁguration ﬁle, by default storage.config. For each valid element in the ﬁle an instance of Span is created. These are of basically four types: • File • Directory • Disk • Raw device After creating all the Span instances, they are grouped by device ID to internal linked lists attached to the Store::disk array[#store-disk-array]_. Spans that refer to the same directory, disk, or raw device are coalesced in to a single span. Spans that refer to the same ﬁle with overlapping offsets are also coalesced5. This is all done in ink_cache_init() called during startup.
: The span logic is also used by the HostDB and more than one otherwise inexplicable feature is provided by the span logic for that module.
After conﬁguration initialization, the cache processor is started by calling CacheProcessor::start(). This does a number of things: For each valid span, an instance of CacheDisk is created. This class is a continuation and so can be used to perform potentially blocking operations on the span. The primary use of these is to be passed to the AIO threads as the callback when an I/O operation completes. These are then dispatched to AIO threads to perform storage unit initialization. After all of those have completed, the resulting storage is distributed across the volumes in cplist_reconfigure(). The CacheVol instances are created at this time. Cache stripe assignment setup is done once all stripes have initialized (that is, the stripe header information has been successfully read from disk for all stripes). The assignment information is stored as an array of indices. These are indices in to an array of stripes. Both the assignment and the stripe arrays are stored in an instance of CacheHostRecord. Assignment initialization consists of populating the assignment array, which is much larger than the stripe array. There is an instance of CacheHostRecord for each line in hosting.config and one generic record. For the conﬁgured instances, the set of stripes is determined from the cache volume speciﬁed in the line. If no lines are
5 This linked list is mostly ignored in later processing, causing all but one ﬁle or directory storage units on the same device to be ignored. See TS-1869.
282 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest speciﬁed, all stripes are placed in the generic record, otherwise only those stripes marked as default are placed in the generic record.
: If hosting records are speciﬁed, it is an error to not specify at least one default cache volume.
The assignment table is initialized in build_vol_hash_table() which is called for each CacheHostRecord instance. For each stripe in the host record, a sequence of pseudo-random numbers is generated. This begins with the folded hash of the stripe hash identiﬁer, which is the device path followed by the skip and size values for that stripe, making it unique. This also makes the sequence deterministic for any particular stripe. Each stripe gets one number in its sequence for every VOL_HASH_ALLOC_SIZE (8 MB currently) of storage. These numbers are paired with the stripe index, combined across all stripes, then sorted by the random values. The resulting array is sampled for every slot in the stripe assignment table by dividing the maximum random value by the size of the assignment table and using the value midway between each multiple of the result of the division. The coalesced psuedo-random sequence is scanned for each sample in turn and the ﬁrst number not greater than the sample is found. The stripe associated with that value is used for that assignment table entry. While this procedure is deterministic, it is sensitive to initial conditions, including the size of each stripe.
Data Structures class OpenDir An open directory entry. It contains all the information of a Dir plus additional information from the ﬁrst Doc. class CacheVC A virtual connection class which accepts input for writing to cache. int CacheVC::openReadStartHead(int event, Event *e) Performs the initial read for a cached object. int CacheVC::openReadStartEarliest(int event, Event *e) Performs the initial read for an alternate of an object. class HttpTunnel Data transfer driver. This contains a set of producers. Each producer is connected to one or more consumers. The tunnel handles events and buffers so that data moves from producers to consumers. The data, as much as possible, is kept in reference counted buffers so that copies are done only when the data is modiﬁed or for sources (which acquire data from outside Trafﬁc Server) and sinks (which move data to outside Trafﬁc Server). class CacheControlResult Holds the data from a line in cache.config. class CacheHTTPInfoVector Deﬁned in P_CacheHttp.h. This is an array of HTTPInfo objects and serves as the respository of informa- tion about alternates of an object. It is marshaled as part of the metadata for an object in the cache. class HTTPInfo Deﬁned in HTTP.h. This class is a wrapper for HTTPCacheAlt. It provides the external API for accessing data in the wrapped class. It contains only a pointer (possibly NULL) to an instance of the wrapped class. class CacheHTTPInfo A typedef for HTTPInfo. class HTTPCacheAlt Deﬁned in HTTP.h. This is the metadata for a single alternate for a cached object. It contains, among other data, the following:
4.6. Cache Architecture 283 Apache-Trafﬁcserver-Server Documentation, latest
•The key for the earliest Doc of the alternate. •The request and response headers. •The fragment offset table.[#fragment-offset-table]_ •Timestamps for request and response from origin server. class EvacuationBlock Record for evacuation. class Vol This represents a storage unit inside a cache volume. off_t Vol::segments The number of segments in the volume. This will be roughly the total number of entries divided by the number of entries in a segment. It will be rounded up to cover all entries. off_t Vol::buckets The number of buckets in the volume. This will be roughly the number of entries in a segment divided by DIR_DEPTH. For currently deﬁned values this is around 16,384 (2^16 / 4). Buckets are used as the targets of the index hash. DLL Vol::evacuate Array of of EvacuationBlock buckets. This is sized so there is one bucket for every evacuation span. off_t len Length of stripe in bytes. int Vol::evac_range(off_t low, off_t high, int evac_phase) Start an evacuation if there is any EvacuationBlock in the range from :arg:‘low‘ to :arg:‘high‘. Return 0 if no evacuation was started, non-zero otherwise. class CacheVol A cache volume as described in volume.config. class Doc Deﬁned in P_CacheVol.h. uint32_t Doc::magic Validity check value. Set to DOC_MAGIC for a valid document. uint32_t Doc::len The length of this segment including the header length, fragment table, and this structure. uint64_t Doc::total_len Total length of the entire document not including meta data but including headers. INK_MD5 Doc::first_key First index key in the document (the index key used to locate this object in the volume index). INK_MD5 Doc::key The index key for this fragment. Fragment keys are computationally chained so that the key for the next and previous fragments can be computed from this key. uint32_t Doc::hlen Document header (metadata) length. This is not the length of the HTTP headers. uint8_t Doc::ftype Fragment type. Currently only CACHE_FRAG_TYPE_HTTP is used. Other types may be used for cache extensions if those are ever implemented. uint24_t Doc::flen Fragment table length, if any. Only the ﬁrst Doc in an object should contain a fragment table.
284 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
The fragment table is a list of offsets relative to the HTTP content (not counting metadata or HTTP head- ers). Each offset is the byte offset of the ﬁrst byte in the fragment. The ﬁrst element in the table is the second fragment (what would be index 1 for an array). The offset for the ﬁrst fragment is of course always zero and so not stored. The purpose of this is to enable a fast seek for range requests. Given the ﬁrst Doc the fragment containing the ﬁrst byte in the range can be computed and loaded directly without further disk access. Removed as of version 3.3.0. uint32_t Doc::sync_serial Unknown. uint32_t Doc::write_serial Unknown. uint32_t pinned Flag and timer for pinned objects. uint32_t checksum Unknown. class VolHeaderFooter
API functions
 void TSHttpTxnReqCacheableSet(TSHttpTxn txnp, int ﬂag) Set a ﬂag that marks a request as cacheable. This is a positive override only, setting :arg:‘ﬂag‘ to 0 restores the default behavior, it does not force the request to be uncacheable. TSReturnCode TSCacheUrlSet(TSHttpTxn txnp, char const* url, int length) Set the cache key for the transaction :arg:‘txnp‘ as the string pointed at by :arg:‘url‘ of :arg:‘length‘ char- acters. It need not be NUL-terminated. This should be called from TS_HTTP_READ_REQUEST_HDR_HOOK which is before cache lookup but late enough that the HTTP request header is available. TSReturnCode TSHttpTxnCacheLookupUrlGet(TSHttpTxn txnp, TSMBuffer bufp, TSMLoc offset) Get the current cache key URL, also referred to as the lookup URL. This must be stored in a properly allocated URL object, typically created with a TSUrlCreate(). TSReturnCode TSHttpTxnCacheLookupUrlSet(TSHttpTxn txnp, TSMBuffer bufp, TSMLoc offset) Set the current cache key URL, also referred to as the lookup URL. This must be stored in a properly allocated URL object, typically created with a TSUrlCreate() or TSUrlClone(). The APIs that modify the cache key can be called as early as TS_HTTP_READ_REQUEST_HDR_HOOK but no later than TS_HTTP_POST_REMAP_HOOK. The cache key is not only used for a cache lookup before going to origin, but also to mark the intent to write to cache on an origin response (if possible).
Cache Internals
 int DIR_SIZE_WITH_BLOCK(int big) A preprocessor macro which computes the maximum size of a fragment based on the value of :arg:‘big‘. This is computed as if the argument where the value of the :arg:‘big‘ ﬁeld in a struct Dir. int DIR_BLOCK_SIZE(int big) A preprocessor macro which computes the block size multiplier for a struct Dir where :arg:‘big‘ is the :arg:‘big‘ ﬁeld value.
4.6. Cache Architecture 285 Apache-Trafﬁcserver-Server Documentation, latest
Cache Tools
Tools and techniques for cache monitoring and inspection. • The cache inspector.
Topics to be done
• Resident alternates • Object refresh
Cache Consistency
The cache is completely consistent, up to and including kicking the power cord out, if the write buffer on consumer disk drives is disabled. You need to use:
 hdparm-W0
The cache validates that all the data for the document is available and will silently mark a partial document as a miss on read. There is no gentle shutdown for Trafﬁc Server. You simply kill the process and the recovery code (fsck) is run every time Trafﬁc Server starts up. On startup the two versions of the index are checked, and the last valid one is read into memory. Trafﬁc Server then moves forward from the last snapped write cursor and reads all the fragments written to disk and updates the directory (as in a log-based ﬁle system). It stops reading at the write before the last valid write header it sees (as a write is not necessarily atomic because of sector reordering). Then the new updated index is written to the invalid version (in case of a crash during startup) and the system starts.
Volume Tagging
Currently, cache volumes are allocated somewhat arbitrarily from storage elements. This enhancement allows storage.config to assign storage units to speciﬁc volumes although the volumes must still be listed in volume. config in general and in particular to map domains to speciﬁc volumes. A primary use case for this is to be able to map speciﬁc types of content to different storage elements. This can be employed to have different storage devices for various types of content (SSD vs. rotational).
Version Upgrade
It is currently the case that any change to the cache format will clear the cache. This is an issue when upgrading the Trafﬁc Server version and should be kept in mind.
Controlling the cache key
The cache key is by default the URL of the request. There are two possible choices, the original (pris- tine) URL and the remapped URL. Which of these is used is determined by the conﬁguration value :ts:cv:‘proxy.conﬁg.url_remap.pristine_host_hdr‘. This is an INT value. If set to 0 (disabled) then the remapped URL is used, and if it is not 0 (enabled) then the original URL is used. This setting also controls the value of the HOST header that is placed in the request sent to the origin server, using the hostname from the original URL if not 0 and the host name from the remapped URL if 0. It has no other effects.
286 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
For caching, this setting is irrelevant if no remapping is done or there is a one-to-one mapping between the original and remapped URLs. It becomes signiﬁcant if multiple original URLs are mapped to the same remapped URL. If pristine headers are enabled, requests to different original URLs will be stored as distinct objects in the cache. If disabled, the remapped URL will be used and there may be collisions. This is bad if the contents different, but quite useful if they are the same (as in situations where the original URLs are just aliases for the same underlying server resource). This is also an issue if a remapping is changed because it is effectively a time axis version of the previous case. If an original URL is remapped to a different server address then the setting determines if existing cached objects will be served for new requests (enabled) or not (disabled). Similarly, if the original URL mapped to a particular URL is changed then cached objects from the initial original URL will be served from the updated original URL if pristine headers is disabled. These collisions are not by themselves good or bad. An administrator needs to decide which is appropriate for their situation and set the value correspondingly. If a greater degree of control is desired, a plugin must be used to invoke the API calls TSHttpTxnCacheLookupUrlSet() or TSCacheUrlSet() to provide a speciﬁc cache key. The TSCacheUrlSet() API can be called as early as TS_HTTP_READ_REQUEST_HDR_HOOK but no later than TS_HTTP_POST_REMAP_HOOK. It can be called only once per transaction; calling it multiple times has no additional effect. A plugin that changes the cache key must do so consistently for both cache hit and cache miss requests because two different requests that map to the same cache key will be considered equivalent by the cache. Use of the URL directly provides this and so must any substitute. This is entirely the responsibility of the plugin; there is no way for the Trafﬁc Server core to detect such an occurrence. If TSHttpTxnCacheLookupUrlGet() is called after new cache url set by TSHttpTxnCacheLookupUrlSet() or TSCacheUrlSet(), it should use a URL location created by TSUrlCreate() as its third input parameter instead of getting url_loc from the client request. It is a requirement that the string be syntactically a URL but otherwise it is completely arbitrary and need not have any path. For instance, if the company Network Geographics wanted to store certain content under its own cache key, using a document GUID as part of the key, it could use a cache key like
 ngeo://W39WaGTPnvg
The scheme ngeo was picked speciﬁcally because it is not a valid URL scheme, and so will never collide with any valid URL. This can be useful if the URL encodes both important and unimportant data. Instead of storing potentially identical content under different URLs (because they differ on the unimportant parts) a url containing only the important parts could be created and used. For example, suppose the URL for Network Geographics content encoded both the document GUID and a referral key.
 http://network-geographics-farm-1.com/doc/W39WaGTPnvg.2511635.UQB_zCc8B8H
We don’t want to serve the same content for every possible referrer. Instead, we could use a plugin to convert this to the previous example and requests that differed only in the referrer key would all reference the same cache entry. Note that we would also map the following to the same cache key
 http://network-geographics-farm-56.com/doc/W39WaGTPnvg.2511635.UQB_zCc8B8H
This can be handy for sharing content between servers when that content is identical. Plugins can change the cache key, or not, depending on any data in the request header. For instance, not changing the cache key if the request is not
4.6. Cache Architecture 287 Apache-Trafﬁcserver-Server Documentation, latest in the doc directory. If distinguishing servers is important, that can easily be pulled from the request URL and used in the synthetic cache key. The implementor is free to extract all relevant elements for use in the cache key. While there is no explicit requirement that the synthetic cache key be based on the HTTP request header, in practice it is generally necessary due to the consistency requirement. Because cache lookup happens before attempting to connect to the origin server, no data from the HTTP response header is available, leaving only the request header. The most common case is the one described above where the goal is to elide elements of the URL that do not affect the content to minimize cache footprint and improve cache hit rates.
RAM Cache
New RAM Cache Algorithm (CLFUS)
The new RAM Cache uses ideas from a number of cache replacement policies and algorithms, including LRU, LFU, CLOCK, GDFS and 2Q, called CLFUS (Clocked Least Frequently Used by Size). It avoids any patented algorithms and includes the following features: • Balances Recentness, Frequency and Size to maximize hit rate (not byte hit rate). • Is Scan Resistant and extracts robust hit rates even when the working set does not ﬁt in the RAM Cache. • Supports compression at 3 levels: fastlz, gzip (libz), and xz (liblzma). Compression can be moved to another thread. • Has very low CPU overhead, only slightly more than a basic LRU. Rather than using an O(lg n) heap, it uses a probabilistic replacement policy for O(1) cost with low C. • Has relatively low memory overhead of approximately 200 bytes per object in memory. The rationale for emphasizing hit rate over byte hit rate is that the overhead of pulling more bytes from secondary storage is low compared to the cost of a request. The RAM Cache consists of an object hash fronting 2 LRU/CLOCK lists and a seen hash table. The ﬁrst cached list contains objects in memory, while the second contains a history of objects which have either recently been in memory or are being considered for keeping in memory. The seen hash table is used to make the algorithm scan resistant. The list entries record the following information: Value Description key 16 byte unique object identiﬁer auxkeys 8 bytes worth of version number (in our system, the block in the partition). When the version of an object changes old entries are purged from the cache. hits Number of hits within this clock period. size size of the object in the cache. len Length of the object, which differs from size because of compression and padding). com- Compressed length of the object. pressed_len com- Compression type, or none if no compression. Possible types are: fastlz, libz, and liblzma. pressed uncom- Flag indicating that content cannot be compressed (true), or that it mat be compressed (false). pressible copy Whether or not this object should be copied in and copied out (e.g. HTTP HDR). LRU link HASH link IOBuffer- Smart point to the data buffer. Data
288 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
The interface to the cache is Get and Put operations. Get operations check if an object is in the cache and are called on a read attempt. The Put operation decides whether or not to cache the provided object in memory. It is called after a read from secondary storage.
Seen Hash
The Seen List becomes active after the Cached and History lists become full following a cold start. The purpose is to make the cache scan resistant, which means that the cache state must not be affected at all by a long sequence Get and Put operations on objects which are seen only once. This is essential, and without it not only would the cache be polluted, but it could lose critical information about the objects that it cares about. It is therefore essential that the Cache and History lists are not affected by Get or Put operations on objects seen the ﬁrst time. The Seen Hash maintains a set of 16 bit hash tags, and requests which do not hit in the object cache (are in the Cache List or History List) and do not match the hash tag result in the hash tag being updated but are otherwise ignored. The Seen Hash is sized to approximately the number of objects in the cache in order to match the number that are passed through it with the CLOCK rate of the Cached and History Lists.
Cached List
The Cached List contains objects actually in memory. The basic operation is LRU with new entries inserted into a FIFO queue and hits causing objects to be reinserted. The interesting bit comes when an object is being considered for insertion. A check is ﬁrst made against the Object Hash to see if the object is in the Cached List or History. Hits result in updating the hit ﬁeld and reinsertion of the object. History hits result in the hit ﬁeld being updated and a comparison to see if this object should be kept in memory. The comparison is against the least recently used members of the Cache List, and is based on a weighted frequency:
CACHE_VALUE= hits/ (size+ overhead)
A new object must be enough bytes worth of currently cached objects to cover itself. Each time an object is considered for replacement the CLOCK moves forward. If the History object has a greater value then it is inserted into the Cached List and the replaced objects are removed from memory and their list entries are inserted into the History List. If the History object has a lesser value it is reinserted into the History List. Objects considered for replacement (at least one) but not replaced have their hits ﬁeld set to 0 and are reinserted into the Cached List. This is the CLOCK operation on the Cached List.
History List
Each CLOCK, the least recently used entry in the History List is dequeued and if the hits ﬁeld is not greater than 1 (it was hit at least once in the History or Cached List) it is deleted. Otherwise, the hits is set to 0 and it is requeued on the History List.
Compression and Decompression
Compression is performed by a background operation (currently called as part of Put) which maintains a pointer into the Cached List and runs toward the head compressing entries. Decompression occurs on demand during a Get. In the case of objects tagged copy, the compressed version is reinserted in the LRU since we need to make a copy anyway. Those not tagged copy are inserted uncompressed in the hope that they can be reused in uncompressed form. This is a compile time option and may be something we want to change. There are 3 algorithms and levels of compression (speed on an Intel i7 920 series processor using one thread):
4.6. Cache Architecture 289 Apache-Trafﬁcserver-Server Documentation, latest
Method Compression Decompression Notes Rate Rate fastlz 173 MB/sec 442 MB/sec Basically free since disk or network will limit ﬁrst; ~53% ﬁnal size. libz 55 MB/sec 234 MB/sec Almost free, particularly decompression; ~37% ﬁnal size. li- 3 MB/sec 50 MB/sec Expensive; ~27% ﬁnal size. blzma These are ballpark numbers, and your millage will vary enormously. JPEG, for example, will not compress with any of these (or at least will only do so at such a marginal level that the cost of compression and decompression is wholly unjustiﬁed), and the same is true of many other media and binary ﬁle types which embed some form of compression. The RAM Cache does detect compression level and will declare something incompressible if it doesn’t get below 90% of the original size. This value is cached so that the RAM Cache will not attempt to compress it again (at least as long as it is in the history).
Tiered Storage
Tiered storage is an attempt to allow Trafﬁc Server to take advantage of physical storage with different properties. This design concerns only mechanism. Policies to take advantage of these are outside of the scope of this document. Instead we will presume an oracle which implements this policy and describe the queries that must be answered by the oracle and the effects of the answers. Beyond avoiding question of tier policy, the design is also intended to be effectively identical to current operations for the case where there is only one tier. The most common case for tiers is an ordered list of tiers, where higher tiers are presumed faster but more expensive (or more limited in capacity). This is not required. It might be that different tiers are differentiated by other properties (such as expected persistence). The design here is intended to handle both cases. The design presumes that if a user has multiple tiers of storage and an ordering for those tiers, they will usually want content stored at one tier level to also be stored at every other lower level as well, so that it does not have to be copied if evicted from a higher tier.
Conﬁguration
Each storage unit in storage.config can be marked with a quality value which is 32 bit number. Storage units that are not marked are all assigned the same value which is guaranteed to be distinct from all explicit values. The quality value is arbitrary from the point of view of this design, serving as a tag rather than a numeric value. The user (via the oracle) can impose what ever additional meaning is useful on this value (rating, bit slicing, etc.). In such cases, all volumes should be explicitly assigned a value, as the default (unmarked) value is not guaranteed to have any relationship to explicit values. The unmarked value is intended to be useful in situations where the user has no interest in tiered storage and so wants to let Trafﬁc Server automatically handle all volumes as a single tier.
Operations
After a client request is received and processed, volume assignment is done. For each tier, the oracle would return one of four values along with a volume pointer: READ The tier appears to have the object and can serve it. WRITE The object is not in this tier and should be written to this tier if possible. RW Treat as READ if possible, but if the object turns out to not in the cache treat as WRITE. NO_SALE Do not interact with this tier for this object.
290 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
The volume returned for the tier must be a volume with the corresponding tier quality value. In effect, the current style of volume assignment is done for each tier, by assigning one volume out of all of the volumes of the same quality and returning one of RW or WRITE, depending on whether the initial volume directory lookup succeeds. Note that as with current volume assignment, it is presumed this can be done from in memory structures (no disk I/O required). If the oracle returns READ or RW for more than one tier, it must also return an ordering for those tiers (it may return an ordering for all tiers, ones that are not readable will be ignored). For each tier, in that order, a read of cache storage is attempted for the object. A successful read locks that tier as the provider of cached content. If no tier has a successful read, or no tier is marked READ or RW then it is a cache miss. Any tier marked RW that fails the read test is demoted to WRITE. If the object is cached, every tier that returns WRITE receives the object to store in the selected volume (this includes RW returns that are demoted to WRITE). This is a cache to cache copy, not from the origin server. In this case, tiers marked RW that are not tested for read will not receive any data and will not be further involved in the request processing. For a cache miss, all tiers marked WRITE will receive data from the origin server connection (if successful). This means, among other things, that if there is a tier with the object all other tiers that are written will get a local copy of the object, and the origin server will not be used. In terms of implementation, currently a cache write to a volume is done via the construction of an instance of CacheVC which recieves the object stream. For tiered storage, the same thing is done for each target volume. For cache volume overrides (via hosting.config) this same process is used except with only the volumes stripes contained within the speciﬁed cache volume.
Copying
It may be necessary to provide a mechanism to copy objects between tiers outside of a client originated transaction. In terms of implementation, this is straight forward using HttpTunnel as if in a transaction, only using a CacheVC instance for both the producer and consumer. The more difﬁcult question is what event would trigger a possible copy. A signal could be provided whenever a volume directory entry is deleted, although it should be noted that the object in question may have already been evicted when this event happens.
Additional Notes
As an example use, it would be possible to have only one cache volume that uses tiered storage for a particular set of domains using volume tagging. hosting.config would be used to direct those domains to the selected cache volume. The oracle would check the URL in parallel and return NO_SALE for the tiers in the target cache volume for other domains. For the other tier (that of the unmarked storage units), the oracle would return RW for the tier in all cases as that tier would not be queried for the target domains.
Plugin Development
Plugin Development Introduction
This chapter provides a foundation for designing and writing plugins. Reading this chapter will help you to understand: • The asynchronous event mode. This is the design paradigm used throughout Trafﬁc Server; plugins must also follow this design. It includes the callback mechanism for Trafﬁc Server to “wake up” your plugin and put it to work. • Trafﬁc Server’s HTTP processing, with an overview of the HTTP state machine. • How plugins can hook onto and modify/extend Trafﬁc Server’s HTTP processing.
4.7. Plugin Development 291 Apache-Trafﬁcserver-Server Documentation, latest
•A Roadmap with an overview of the functionality provided by the Trafﬁc Server API.
Roadmap
This chapter has provided an overview of Trafﬁc Server’s HTTP processing, API hooks, and the asynchronous event model. Next, you must understand the capabilities of Trafﬁc Server API functions. These are quite broad: • HTTP header manipulation functions Obtain information about and manipulate HTTP headers, URLs, & MIME headers. • HTTP transaction functions Get information about and modify HTTP transactions (for example: get the client IP associated to the transac- tion; get the server IP; get parent proxy information) • IO functions Manipulate vconnections (virtual connections, used for network and disk I/O) • Network connection functions Open connections to remote servers. • Statistics functions Deﬁne and compute statistics for your plugin’s activity. • Trafﬁc Server management functions Obtain values for Trafﬁc Server conﬁguration and statistics variables. Below are some guidelines for creating a plugin: 1. Decide what you want your plugin to do, based on the capabilities of the API and Trafﬁc Server. The two kinds of example plugins provided with this SDK are HTTP-based (includes header-based and response transform plugins), and non-HTTP-based (a protocol plugin). These examples are discussed in the following chapters. 2. Determine where your plugin needs to hook on to Trafﬁc Server’s HTTP processing (view the HTTP Transaction State Diagram. 3. Read Header-Based Plugin Examples to learn the basics of writing plugins: creating continuations and setting up hooks. If you want to write a plugin that transforms data, then read HTTP Transformations. 4. Figure out what parts of the Trafﬁc Server API you need to use and then read about the details of those APIs in this manual’s reference chapters. 5. Compile and load your plugin (see Getting Started. 6. Depending on your plugin’s functionality, you might start testing it by issuing requests by hand and checking for the desired behavior in Trafﬁc Server log ﬁles. See the *Trafﬁc Server Administrator’s Guide* for information about Trafﬁc Server logs.
Asynchronous Event Model
Trafﬁc Server is a multi-threaded process. There are two main reasons why a server might use multiple threads: • To take advantage of the concurrency available with multiple CPUs and multiple I/O devices. • To manage concurrency from having many simultaneous client connections. For example, a server could create one thread for each connection, allowing the operating system (OS) to control switching between threads.
292 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Trafﬁc Server uses multiple threads for the ﬁrst reason. However, Trafﬁc Server does not use a separate OS thread per transaction because it would not be efﬁcient when handling thousands of simultaneous connections. Instead, Trafﬁc Server provides special event-driven mechanisms for efﬁciently scheduling work: the event system and continuations. The event system is used to schedule work to be done on threads. A continuation is a passive, event- driven state machine that can do some work until it reaches a waiting point; it then sleeps until it receives notiﬁcation that conditions are right for doing more work. For example, HTTP state machines (which handle HTTP transactions) are implemented as continuations. Continuation objects are used throughout Trafﬁc Server. Some might live for the duration of the Trafﬁc Server process, while others are created (perhaps by other continuations) for speciﬁc needs and then destroyed. Trafﬁc Server Internals (below) shows how the major components of Trafﬁc Server interact. Trafﬁc Server has several processors, such as cache processor and net processor, that consolidate cache or network I/O tasks. Processors talk to the event system and schedule work on threads. An executing thread calls back a continuation by sending it an event. When a continuation receives an event, it wakes up, does some work, and either destroys itself or goes back to sleep & waits for the next event. Trafﬁc Server Internals
4.6: Trafﬁc Server Internals
Plugins are typically implemented as continuations. All of the sample code plugins (except hello-world) are continuations that are created when Trafﬁc Server starts up; they then wait for events that trigger them into activity. Trafﬁc Server with Plugins A plugin may consist of just one static continuation that is called whenever certain events happen. Examples of such plugins include blacklist-1.c, basic-auth.c, and redirect-1.c. Alternatively, a plugin might dynam- ically create other continuations as needed. Transform plugins are built in this manner: a static parent continuation checks all transactions to see if any are transformable; when a transaction is transformable, the static continuation creates a type of continuation called a vconnection. The vconnection lives as long as it takes to complete the trans- form and then destroys itself. This design can be seen in all of the sample transform plugins. Plugins that support new protocols also have this architecture: a static continuation listens for incoming client connections and then creates transaction state machines to handle each protocol transaction.
4.7. Plugin Development 293 Apache-Trafﬁcserver-Server Documentation, latest
4.7: Trafﬁc Server with Plugins
When you write plugins, there are several ways to send events to continuations. For HTTP plugins, there is a “hook” mechanism that enables the Trafﬁc Server HTTP state machine to send your plugin wakeup calls when needed. Additionally, several Trafﬁc Server API functions trigger Trafﬁc Server sub-processes to send events to plugins: TSContCall, TSVConnRead, TSCacheWrite, and TSMgmtUpdateRegister, to name a few.
Naming Conventions
Trafﬁc Server HTTP State Machine
Trafﬁc Server performs sophisticated HTTP caching and proxying. Important features include checking for alternates and document freshness, ﬁltering, supporting cache hierarchies, and hosting. Trafﬁc Server handles thousands of client requests at a time and each request is handled by an HTTP state machine. These machines follow a complex state diagram that includes all of the states required to support Trafﬁc Server’s features. The Trafﬁc Server API provides hooks to a subset of these states, chosen for their relevance to plugins. You can view the API hooks and corresponding HTTP states in the HTTP Transaction State Diagram. The example in this section (below) explains how a plugin typically intervenes and extends Trafﬁc Server’s processing of an HTTP transaction. Complete details about hooking on to Trafﬁc Server processes are provided in Hooks and Transactions.
HTTP Transaction
An HTTP transaction consists of a client request for a web document and Trafﬁc Server’s response. The response could be the requested web server content or it could be an error message. The content could come from the Trafﬁc Server cache or Trafﬁc Server might fetch it from the origin server. The following diagram shows some states in a typical transaction - speciﬁcally, the scenario wherein content is served from cache. Simpliﬁed HTTP Transaction
294 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
4.8: Simpliﬁed HTTP Transaction
In the diagram above, Trafﬁc Server accepts the client connection, reads the request headers, looks up the origin server’s IP address, and looks for the requested content in the cache. If the content is not in the cache (a “miss”), then Trafﬁc Server opens a connection to the origin server and issues a request for the content. If the content is in the cache (a “hit”), then Trafﬁc Server checks it for freshness. If the content is fresh, then Trafﬁc Server sends a reply header to the client. If the content is stale, then Trafﬁc Server opens a connection to the origin server and requests the content. The ﬁgure above, Simpliﬁed HTTP Transaction, does not show behavior in the event of an error. If there is an error at a any stage, then the HTTP state machine jumps to the “send reply header” state and sends a reply. If the reply is an error, then the transaction closes. If the reply is not an error, then Trafﬁc Server ﬁrst sends the response content before it closes the transaction. API Hooks Corresponding to States
4.9: API Hooks Corresponding to States Listed in
You use hooks as triggers to start your plugin. The name of a hook reﬂects the Trafﬁc Server state that was just completed. For example, the “OS DNS lookup” hook wakes up a plugin right after the origin server DNS lookup. For a plugin that requires the IP address of the requested origin server, this hook is the right one to use. The Blacklist
4.7. Plugin Development 295 Apache-Trafﬁcserver-Server Documentation, latest
 plugin works in this manner, as shown in the Blacklist Plugin diagram below. Blacklist Plugin
4.10: Blacklist Plugin
Trafﬁc Server calls the Blacklist plugin right after the origin server DNS lookup. The plugin checks the requested host against a list of blacklisted servers; if the request is allowed, then the transaction proceeds. If the host is forbidden, then the Blacklist plugin sends the transaction into an error state. When the HTTP state machine gets to the “send reply header” state, it then calls the Blacklist plugin to provide the error message that’s sent to the client.
Types of Hooks
The Blacklist plugin’s hook to the origin server DNS lookup state is a global hook, meaning that the plugin is called every time there’s an HTTP transaction with a DNS lookup event. The plugin’s hook to the send reply header state is a transaction hook, meaning that this hook is only invoked for speciﬁed transactions (in the Blacklist Plugin example, it’s only used for requests to blacklisted servers). Several examples of setting up hooks are provided in Header-Based Plugin Examples and HTTP Transformations. Header manipulation plugins, such as ﬁltering, basic authorization, or redirects, usually have a global hook to the DNS lookup or the read request header states. If speciﬁc actions need to be done to the transaction further on, then the plugin adds itself to a transaction hook. Transformation plugins require a global hook to check all transactions for transformability followed by a transform hook, which is a type of transaction hook used speciﬁcally for transforms.
Getting Started
A Simple Plugin
This section describes how to write, compile, conﬁgure, and run a simple Trafﬁc Server plugin. You’ll follow the steps below:
296 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
1. Make sure that your plugin source code contains an TSPluginInit initialization function. 2. Compile your plugin source code, creating a shared library. 3. Add an entry to your plugin’s plugin.config ﬁle. 4. Add the path to your plugin shared library into the records.config ﬁle. 5. Restart Trafﬁc Server.
Compile Your Plugin
The process for compiling a shared library varies with the platform used, so the Trafﬁc Server API provides the tsxs tool which you can use to create shared libraries on all the supported Trafﬁc Server platforms.
Example
Assuming the sample program is stored in the ﬁle hello-world.c, you could use the following commands to build a shared library
 tsxs-o hello-world.so-c hello-world.c
 tsxs can be found in trafficserver-dev package. This shared library will be your plugin. In order to install it, run
 sudo tsxs-o hello-world.so-i
 or the equivalent to sudo on your platform.
Update the plugin.config File
Your next step is to tell Trafﬁc Server about the plugin by adding the following line to the plugin.config ﬁle. Since our simple plugin does not require any arguments, the following plugin.config will work:
# a simple plugin.config for hello-world hello-world.so
Trafﬁc Server can accommodate multiple plugins. If several plugin functions are triggered by the same event, then Trafﬁc Server invokes each plugin’s function in the order each was deﬁned in the plugin.config ﬁle.
Specify the Plugin’s Location
All plugins must be located in the directory speciﬁed by the conﬁguration variable proxy.config.plugin. plugin_dir, which is located in the records.config ﬁle. The directory can be speciﬁed as an absolute or relative path. If a relative path is used, then the starting directory will be the Trafﬁc Server installation directory as speciﬁed in /etc/traffic_server. The default value is libexec/trafficserver, but this can vary based on how the software was conﬁgured and built. It is common to use the default directory. Be sure to place the shared library hello-world.so inside the directory you’ve conﬁgured.
4.7. Plugin Development 297 Apache-Trafﬁcserver-Server Documentation, latest
Restart Trafﬁc Server
The last step is to start/restart Trafﬁc Server. Shown below is the output displayed after you’ve created and loaded your hello-world plugin.
# ls libexec/trafficserver hello-world.so* # bin/traffic_server [Mar 27 19:06:31.669] NOTE: updated diags config [Mar 27 19:06:31.680] NOTE: loading plugin'libexec/trafficserver/hello-world.so' hello world [Mar 27 19:06:32.046] NOTE: cache disabled (initializing) [Mar 27 19:06:32.053] NOTE: cache enabled [Mar 27 19:06:32.526] NOTE: Traffic Server running
Note: in the example above, Trafﬁc Server notes are directed to the console by specifying E for proxy.config. diags.output.note in records.config. The second note shows Trafﬁc Server attempting to load the hello-world plugin. The third line of Trafﬁc Server output is from your plugin.
Plugin Registration and Version Checking
If you need you make a plugin that will load against multiple versions of Trafﬁc Server, you can check the API version at both compilation time and run time. Use the following interfaces: • TSPluginRegister() • TSTrafficServerVersionGet() The plugin registers the plugin and ensures it’s running with a compatible version of Trafﬁc Server.
Naming Conventions
The Trafﬁc Server API adheres to the following naming conventions: • The TS preﬁx is used for all function and variable names deﬁned in the Trafﬁc Server API. Examples: TS_EVENT_NONE,TSMutex, and TSContCreate • Enumerated values are always written in all uppercase letters. Examples: TS_EVENT_NONE and TS_VC_CLOSE_ABORT • Constant values are all uppercase; enumerated values can be seen as a subset of constants. Examples: TS_URL_SCHEME_FILE and TS_MIME_FIELD_ACCEPT • The names of deﬁned types are mixed-case. Examples: TSHttpSsn and TSHttpTxn • Function names are mixed-case. Examples: TSUrlCreate and TSContDestroy • Function names use the following subject-verb naming style: TS--, where  goes from general to speciﬁc. This makes it easier to determine what a function does by reading its name. For example: the function to retrieve the password ﬁeld (the speciﬁc subject) from a URL (the general subject) is TSUrlPasswordGet. • Common verbs like Create, Destroy, Get, Set, Copy, Find, Retrieve, Insert, Remove, and Delete are used only when appropriate. The Trafﬁc Server API enables you to create plugins, using the C programming language, that customize the behavior of your Trafﬁc Server installation. This chapter contains the following sections:
298 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
• Understanding Trafﬁc Server Plugins – a brief introduction to plugins. • A Simple Plugin – walks through compiling and loading an example hello world plugin. • Plugin Registration and Version Checking – shows you how to register your plugin and make sure it’s compatible with the version of Trafﬁc Server you’re using. • Naming Conventions – outlines Trafﬁc Server API naming conventions. For guidelines on creating plugin source code, see Plugin Development Introduction.
Understanding Trafﬁc Server Plugins
Trafﬁc Server enables sophisticated caching and processing of web-related trafﬁc, such as DNS and HTTP requests and responses. Trafﬁc Server itself consists of an event-driven loop that can be simpliﬁed as follows:
 for (;;) { event= get_next_event(); handle_event (event); }
The Role of Plugins
You compile your plugin source code to create a shared library that Trafﬁc Server loads when it is started. Your plugin contains callback functions that are registered for speciﬁc Trafﬁc Server events. When Trafﬁc Server needs to process an event, it invokes any and all call-back functions you’ve registered for that event type.
: Since plugins add object code to Trafﬁc Server, programming errors in a plugin can have serious implications. Bugs in your plugin, such as an out-of-range pointer, can cause Trafﬁc Server processes to crash and may ultimately result in unpredictable behavior.
Plugin Process
Possible Uses for Plugins
Possible uses for plugins include the following: • HTTP processing: plugins can ﬁlter, blacklist, authorize users, transform content • Protocol support: plugins can enable Trafﬁc Server to proxy-cache new protocol content Some examples of plugins include: • Blacklisting plugin: denies attempts to access web sites that are off-limits. • Append transform plugin: adds text to HTTP response content. • Image conversion plugin: transforms JPEG images to GIF images. • Compression plugin: sends response content to a compression server that compresses the data (alternatively, a compression library local to the Trafﬁc Server host machine could do the compression). • Authorization plugin: checks a user’s permissions to access particular web sites. The plugin could consult a local authorization program or send queries to an authorization server.
4.7. Plugin Development 299 Apache-Trafﬁcserver-Server Documentation, latest
4.11: Plugin Process
• A plugin that gathers client information from request headers and enters this information in a database. • Protocol plugin: listens for speciﬁc protocol requests on a designated port and then uses Trafﬁc Server’s proxy server & cache to serve client requests. The ﬁgure below, Possible Trafﬁc Server Plugins, illustrates several types of plugins. Possible Trafﬁc Server Plugins You can ﬁnd basic examples for many plugins in the SDK sample code: • append-transform.c adds text from a speciﬁed ﬁle to HTTP/text responses. This plugin is explained in Append-Transform Plugin • The compression plugin in the ﬁgure communicates with the server that actually does the compression. The server-transform.c plugin shows how to open a connection to a transformation server, have the server do the transformation, and send transformed data back to the client. Although the transformation is null in server-transform.c, a compression or image translation plugin could be implemented in a similar way. • basic-auth.c performs basic HTTP proxy authorization. • blacklist-1.c reads blacklisted servers from a conﬁguration ﬁle and denies client access to these servers. This plugin is explained in Blacklist Plugin.
Plugin Loading
When Trafﬁc Server is ﬁrst started, it consults the plugin.config ﬁle to determine the names of all shared plugin libraries that need to be loaded. The plugin.config ﬁle also deﬁnes arguments that are to be passed to each plugin’s initialization function, TSPluginInit. The records.config ﬁle deﬁnes the path to each plugin shared library, as described in Specify the Plugin’s Location.
300 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
4.12: Possible Trafﬁc Server Plugins
4.7. Plugin Development 301 Apache-Trafﬁcserver-Server Documentation, latest
: The path for each of these ﬁles is /config/, where  is where you installed Trafﬁc Server.
Plugin Conﬁguration
The sample plugin.config ﬁle below contains a comment line, a blank line, and two plugin conﬁgurations:
# This is a comment line. my-plugin.so junk.example.com trash.example.org garbage.example.edu some-plugin.so arg1 arg2 $proxy.config.http.cache.on
Each plugin conﬁguration in the plugin.config ﬁle resembles a UNIX or DOS shell command; each line in plugin.config cannot exceed 1023 characters. The ﬁrst plugin conﬁguration is for a plugin named my-plugin.so. It contains three arguments that are to be passed to that plugin’s initialization routine. The second conﬁguration is for a plugin named some-plugin.so; it contains three arguments. The last argument, ‘‘$proxy.conﬁg.http.cache.on‘‘, is actually a conﬁguration variable. Trafﬁc Server will look up the speciﬁed conﬁguration variable and substitute its value. Plugins with global variables should not appear more than once in plugin.config. For example, if you enter: add-header.so header1 add-header.so header2 then the second global variable, header2, will be used for both instances. A simple workaround is to give different names to different instances of the same plugin. For example: cp add-header.so add-header1.so cp add-header.so add-header2.so
These entries will produce the desired result below: add-header1.so header1 add-header2.so header2
Conﬁguration File Rules
• Comment lines begin with # and continue to the end of the line. • Blank lines are ignored. • Plugins are loaded and initialized by Trafﬁc Server in the order they appear in the plugin.config ﬁle.
Plugin Initialization
Each plugin must deﬁne an initialization function named TSPluginInit that Trafﬁc Server invokes when the plugin is loaded. The TSPluginInit function is commonly used to read conﬁguration information and register hooks for event notiﬁcation. The TSPluginInit function has two arguments: • The argc argument represents the number of arguments deﬁned in the plugin.config ﬁle for that particular plugin
302 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
• The argv argument is an array of pointers to the actual arguments deﬁned in the plugin.config ﬁle for that plugin See TSPluginInit() for details about TSPluginInit.
Building Plugins
Plugin Conﬁguration
The functions discussed in this section do not examine or modify Trafﬁc Server conﬁguration variables. To examine Trafﬁc Server conﬁguration and statistics variables, see Settings and Statistics. The collection of TSConfig* functions are designed to provide a fast and efﬁcient mechanism for accessing and changing global conﬁguration information within a plugin. Such a mechanism is simple enough to provide in a single- threaded program, but the translation to a multi-threaded program such as Trafﬁc Server is difﬁcult. A common technique is to have a single mutex protect the global conﬁguration information; however, the problem with this solution is that a single mutex becomes a performance bottleneck very quickly. These functions deﬁne an interface to storing and retrieving an opaque data pointer. Internally, Trafﬁc Server maintains reference count information about the data pointer so that a call to TSConfigSet() will not disturb another thread using the current data pointer. The philosophy is that once a user has a hold of the conﬁguration pointer, it is okay for it to be used even if the conﬁguration changes. All that a user typically wants is a non-changing snapshot of the conﬁguration. You should use TSConfigSet() for all global data updates. Here’s how the interface works:
/* Assume that you have previously defined a plugin configuration * data structure named ConfigData, along with its constructor * plugin_config_allocator () and its destructor * plugin_config_destructor (ConfigData *data) */ ConfigData *plugin_config;
/* You will need to assign plugin_config a unique identifier of type * unsigned int. It is important to initialize this identifier to zero * (see the documentation of the function). */ static unsigned int my_id=0;
/* You will need an TSConfig pointer to access a snapshot of the * current plugin_config. */ TSConfig config_ptr;
/* Initialize plugin_config. */ plugin_config= plugin_config_allocator();
/* Assign plugin_config an identifier using TSConfigSet. */ my_id= TSConfigSet (my_id, plugin_config, plugin_config_destructor);
/* Get a snapshot of the current configuration using TSConfigGet. */ config_ptr= TSConfigGet (my_id);
/* With an TSConfig pointer to the current configuration, you can * retrieve the configuration's current data using TSConfigDataGet. */ plugin_config= (ConfigData *) TSConfigDataGet (config_ptr);
4.7. Plugin Development 303 Apache-Trafﬁcserver-Server Documentation, latest
/* Do something with plugin_config here. */
/* When you are done with retrieving or modifying the plugin data, you * release the pointers to the data with a call to TSConfigRelease. */ TSConfigRelease (my_id, config_ptr);
/* Any time you want to modify plugin_config, you must repeat these * steps, starting with * my_id = TSConfigSet (my_id,plugin_config, plugin_config_destructor); * and continuing up to TSConfigRelease. */
The conﬁguration functions are: • TSConfigDataGet() • TSConfigGet() • TSConfigRelease() • TSConfigSet()
Plugin Management
This chapter covers the following topics:
Settings and Statistics
Your plugin might need to know information about Trafﬁc Server’s current conﬁguration and performance. The func- tions described in this section read this information from the Trafﬁc Server records.config ﬁle. Conﬁguration settings are stored in CONFIG variables and statistics are stored in PROCESS variables.
: Not all CONFIG and PROCESS variables in records.config are relevant to Trafﬁc Server’s conﬁguration and statistics. Therefore, retrieve only the records.config variables that are documented in the Administra- tor’s Guide.
To retrieve a variable, you need to know its type (int, counter, float, or string). Plugins store the records. config values as an TSMgmtInt, TSMgmtCounter, TSMgmtFloat, or TSMgmtString. You can look up records.config variable types in the Administrator’s Guide. Depending on the result type, you’ll use TSMgmtIntGet, TSMgmtCounterGet, TSMgmtFloatGet, or TSMgmtStringGet to obtain the variable value (see the example for TSMgmtIntGet(). The TSMgmt*Get functions are: • TSMgmtCounterGet() • TSMgmtFloatGet() • TSMgmtIntGet() • TSMgmtStringGet()
304 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Logging API
The logging API enables your plugin to log entries in a custom text log ﬁle that you create with the call TSTextLogObjectCreate(). This log ﬁle is part of Trafﬁc Server’s logging system; by default, it is stored in the logging directory. Once you have created the log object, you can set log properties. The logging API enables you to: • Establish a custom text log for your plugin: see TSTextLogObjectCreate() • Set the log header for your custom text log: see TSTextLogObjectHeaderSet() • Enable or disable rolling your custom text log: see TSTextLogObjectRollingEnabledSet() • Set the rolling interval (in seconds) for your custom text log: see TSTextLogObjectRollingIntervalSecSet() • Set the rolling offset for your custom text log: see TSTextLogObjectRollingOffsetHrSet() • Set the rolling size for your custom text log: see TSTextLogObjectRollingSizeMbSet() • Write text entries to the custom text log: see TSTextLogObjectWrite() • Flush the contents of the custom text log’s write buffer to disk: see TSTextLogObjectFlush() • Destroy custom text logs when you are done with them: see TSTextLogObjectDestroy() The steps below show how the logging API is used in the blacklist-1.c sample plugin. For the complete source code, see the Sample Source Code section. 1. A new log ﬁle is deﬁned as a global variable.
 static TSTextLogObject log;
2. In TSPluginInit, a new log object is allocated:
TSReturnCode error= TSTextLogObjectCreate("blacklist", TS_LOG_MODE_ADD_TIMESTAMP,&log);
The new log is named blacklist.log. Each entry written to the log will have a timestamp. The NULL argument speciﬁes that the new log does not have a log header. The error argument stores the result of the log creation; if the log is created successfully, then an error will be equal to TS_LOG_ERROR_NO_ERROR. 3. After creating the log, the plugin makes sure that the log was created successfully:
 if (error!= TS_SUCCESS) { printf("Blacklist plugin: error %d while creating log\n", error); }
4. The Blacklist Plugin matches the host portion of the URL (in each client request) with a list of blacklisted sites (stored in the array sites[]):
 for (i=0; i< nsites; i++){ if (strncmp (host, sites[i], host_length)==0){ /* ... */ } }
If the host matches one of the blacklisted sites (such as sites[i]), then the plugin writes a blacklist entry to blacklist.log:
4.7. Plugin Development 305 Apache-Trafﬁcserver-Server Documentation, latest
 if (log) { TSTextLogObjectWrite(log,"blacklisting site: %s", sites[i]);
The format of the log entry is as follows:
 blacklisting site: sites[i]
The log is not ﬂushed or destroyed in the blacklist-1 plugin - it lives for the life of the plugin.
Actions
Hosts Lookup API
The hosts lookup enables plugins to ask Trafﬁc Server to do a host lookup of a host name, much like a DNS lookup. The hosts lookup functions are as follows: • TSHostLookup() • TSHostLookupResultAddrGet() An action is a handle to an operation initiated by a plugin that has not yet completed. For example: when a plugin connects to a remote server, it uses the call TSNetConnect - which takes TSCont as an argument to call back when the connection is established. TSNetConnect might not call the continuation back immediately and will return an TSAction structure that the caller can use to cancel the operation. Cancelling the operation does not necessarily mean that the operation will not occur; it simply means that the continuation passed into the operation will not be called back. In such an example, the connection might still occur if the action is cancelled; however, the continuation that initiated the connection would not be called back. In the preceding example, it is also possible that the connection will complete and call back the continuation before TSNetConnect returns. If that occurs, then TSNetConnect returns a special action that causes TSActionDone to return 1. This speciﬁes that the operation has already completed, so it’s pointless to try to cancel the operation. Also note that an action will never change from non-completed to completed. When the operation actually succeeds and the continuation is called back, the continuation must zero out its action pointer to indicate to itself that the operation succeeded. The asynchronous nature of all operations in Trafﬁc Server necessitates actions. You should notice from the above discussion that once a call to a function like TSNetConnect is made by a continuation and that function returns a valid action (TSActionDone returns 0), it is not safe for the continuation to do anything else except return from its handler function. It is not safe to modify or examine the continuation’s data because the continuation may have already been destroyed. Below is an example of typical usage for an action:
#include  static int handler (TSCont contp, TSEvent event, void *edata) { if (event== TS_EVENT_IMMEDIATE) { TSAction actionp= TSNetConnect (contp, 127.0.0.1, 9999); if (!TSActionDone (actionp)) { TSContDataSet (contp, actionp); } else { /* We've already been called back... */ return 0; } } else if (event== TS_EVENT_NET_CONNECT) {
306 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
/* Net connection succeeded */ TSContDataSet (contp, NULL); return 0; } else if (event== TS_EVENT_NET_CONNECT_FAILED) { /* Net connection failed */ TSContDataSet (contp, NULL); return 0; } return 0; }
 void TSPluginInit( int argc, const char *argv[]) { TSCont contp;
 contp= TSContCreate (handler, TSMutexCreate ());
/* We don't want to call things out of TSPluginInit directly since it's called before the rest of the system is initialized. We'll simply schedule an event on the continuation to occur as soon as the rest of the system is started up. */ TSContSchedule (contp,0); }
The example above shows a simple plugin that creates a continuation and then schedules it to be called immediately. When the plugin’s handler function is called the ﬁrst time, the event is TS_EVENT_IMMEDIATE. The plugin then tries to open a net connection to port 9999 on localhost (127.0.0.1). The IP description was left in cider notation to further clarify what is going on; also note that the above won’t actually compile until the IP address is modiﬁed. The action returned from TSNetConnect is examined by the plugin. If the operation has not completed, then the plugin stores the action in its continuation. Otherwise, the plugin knows it has already been called back and there is no reason to store the action pointer. A ﬁnal question might be, “why would a plugin want to cancel an action?” In the above example, a valid reason could be to place a limit on the length of time it takes to open a connection. The plugin could schedule itself to get called back in 30 seconds and then initiate the net connection. If the timeout expires ﬁrst, then the plugin would cancel the action. The following sample code implements this:
#include  static int handler (TSCont contp, TSEvent event, void *edata) { switch (event) { case (TS_EVENT_IMMEDIATE): TSContSchedule (contp, 30000); TSAction actionp= TSNetConnect(contp, 127.0.0.1, 9999); if (!TSActionDone (actionp)) { TSContDataSet (contp, actionp); } else { /* We've already been called back ... */ } break;
 case (TS_EVENT_TIMEOUT): TSAction actionp= TSContDataGet (contp); if (!TSActionDone(actionp)) {
4.7. Plugin Development 307 Apache-Trafﬁcserver-Server Documentation, latest
TSActionCancel (actionp); } break;
 case (TS_EVENT_NET_CONNECT): /* Net connection succeeded */ TSContDataSet (contp, NULL); break;
 case (TS_EVENT_NET_CONNECT_FAILED): /* Net connection failed */ TSContDataSet (contp, NULL); break;
} return 0; }
 void TSPluginInit( int argc, const char *argv[]) { TSCont contp;
 contp= TSContCreate (handler, TSMutexCreate ()); /* We don't want to call things out of TSPluginInit directly since it's called before the rest of the system is initialized. We'll simply schedule an event on the continuation to occur as soon as the rest of the system is started up. */ TSContSchedule (contp,0); }
The action functions are: • TSActionCancel() • TSActionDone()
Hooks and Transactions
Hooks are points in Trafﬁc Server transaction processing where plugins can step in and do some work. Registering a plugin function for callback amounts to “adding” the function to a hook. You can register your plugin to be called back for every single transaction or only for speciﬁc transactions. This chapter contains the following sections:
Adding Hooks
There are several ways to add hooks to your plugin. • Global HTTP hooks HTTP transaction hooks are set on a global basis using the function TSHttpHookAdd. This means that the continuation speciﬁed as the parameter to TSHttpHookAdd is called for every transaction. TSHttpHookAdd must be used in TSPluginInit. • Transaction hooks Transaction hooks can be used to call plugins back for a speciﬁc HTTP transaction. You cannot add transaction hooks in TSPluginInit; you ﬁrst need a handle to a transaction. See Accessing the Transaction Being Processed.
308 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
• Transformation hooks Transformation hooks are a special case of transaction hooks. See TSVConnCacheObjectSizeGet() for more information about transformation hooks. You add a trans- formation hook using TSHttpTxnHookAdd, as described in HTTP Transactions. • Session hooks An HTTP session starts when a client opens a connection to Trafﬁc Server and ends when the connection closes. A session can consist of several transactions. Session hooks enable you to hook your plugin to a particular point in every transaction within a speciﬁed session (see HTTP Sessions). Session hooks are added in a manner similar to transaction hooks (ie, you ﬁrst need a handle to an HTTP session). • HTTP select alternate hook Alternate selection hooks enable you to hook on to the alternate selection state. These hooks must be added globally, since Trafﬁc Server does not have a handle to a transaction or session when alternate selection is taking place. See HTTP Alternate Selection for information on the alternate selection mechanism. All of the hook addition functions (TSHttpHookAdd(), TSHttpSsnHookAdd(), TSHttpSsnReenable()) take TSHttpHookID (identiﬁes the hook to add on to) and TSCont (the basic callback mechanism in Trafﬁc Server). A single TSCont can be added to any number of hooks at a time. An HTTP hook is identiﬁed by the enumerated type TSHttpHookID. The values for TSHttpHookID are: Values for TSHttpHookID TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK Called after the HTTP state machine has completed the cache lookup for the document requested in the ongoing transaction. Register this hook via TSHttpTxnHookAdd or TSHttpHookAdd. Corresponds to the event TS_EVENT_HTTP_CACHE_LOOKUP_COMPLETE. TS_HTTP_OS_DNS_HOOK Called immediately after the HTTP state machine has completed a DNS lookup of the origin server. The HTTP state machine will know the origin server’s IP address at this point, which is useful for performing both authentication and blacklisting. Corresponds to the event TS_EVENT_HTTP_OS_DNS. TS_HTTP_POST_REMAP_HOOK Called immediately after remapping occurs, before cache lookup. Corresponds to the event TS_EVENT_HTTP_POST_REMAP. TS_HTTP_PRE_REMAP_HOOK Called after the request header is read from the client, before any remapping of the headers occurs. Corresponds to the event TS_EVENT_HTTP_PRE_REMAP. TS_HTTP_READ_CACHE_HDR_HOOK Called immediately after the request and response header of a previously- cached object is read from cache. This hook is only called if the document is being served from cache. Corre- sponds to the event TS_EVENT_HTTP_READ_CACHE_HDR. TS_HTTP_READ_RESPONSE_HDR_HOOK Called immediately after the response header is read from the origin server or parent proxy. Corresponds to the event TS_EVENT_HTTP_READ_RESPONSE_HDR. TS_HTTP_RESPONSE_TRANSFORM_HOOK See “Transformations” for information about transformation hooks. TS_HTTP_READ_REQUEST_HDR_HOOK Called immediately after the request header is read from the client. Cor- responds to the event TS_EVENT_HTTP_READ_REQUEST_HDR. TS_HTTP_REQUEST_TRANSFORM_HOOK See “Transformations” for information about transformation hooks. TS_HTTP_SELECT_ALT_HOOK See “HTTP Alternate Selection” for information about the alternate selection mechanism. TS_HTTP_SEND_RESPONSE_HDR_HOOK Called immediately before the proxy’s response header is written to the client; this hook is usually used for modifying the response header. Corresponds to the event TS_EVENT_HTTP_SEND_RESPONSE_HDR. TS_HTTP_SEND_REQUEST_HDR_HOOK Called immediately before the proxy’s request header is sent to the origin server or the parent proxy. This hook is not called if the document is being served from cache. This hook is usually used for modifying the proxy’s request header before it is sent to the origin server or parent proxy. TS_HTTP_SSN_CLOSE_HOOK Called when an HTTP session ends. A session ends when the client connection is closed. You can only add this hook as a global hook
4.7. Plugin Development 309 Apache-Trafﬁcserver-Server Documentation, latest
TS_HTTP_SSN_START_HOOK Called when an HTTP session is started. A session starts when a client connects to Trafﬁc Server. You can only add this hook as a global hook. TS_HTTP_TXN_CLOSE_HOOK Called when an HTTP transaction ends. TS_HTTP_TXN_START_HOOK Called when an HTTP transaction is started. A transaction starts when either a client connects to Trafﬁc Server and data is available on the connection, or a previous client connection that was left open for keep alive has new data available. The function you use to add a global HTTP hook is TSHttpHookAdd().
HTTP Sessions
An HTTP session is an object that is deﬁned for the lifetime of a client’s TCP session. The Trafﬁc Server API enables you to add a global hook to the start or end of an HTTP session, as well as add session hooks that call back your plugin for every transaction within a given session. When a client connects to Trafﬁc Server, it opens up a TCP connection and sends one or more HTTP requests. An individual request and its response comprise the HTTP transaction. The HTTP session begins when the client opens the connection and ends when the connection closes. The HTTP session hooks are: • TS_HTTP_SSN_START_HOOK Called when an HTTP session is started (a session starts when a client connects to Trafﬁc Server). This hook must be added as a global hook. • TS_HTTP_SSN_CLOSE_HOOK Called when an HTTP session ends (a session ends when the client connection is closed). This hook must be added as a global hook. Use the session hooks to get a handle to a session (an TSHttpSsn object). If you want your plugin to be called back for each transaction within the session, then use TSHttpSsnHookAdd. Note: you must reenable the session with TSHttpSsnReenable after processing a session hook. The session hook functions are listed below: • TSHttpSsnHookAdd() • TSHttpSsnReenable()
HTTP Transactions
The HTTP transaction functions enable you to set up plugin callbacks to HTTP transactions and obtain/modify infor- mation about particular HTTP transactions. As described in the section on HTTP sessions, an HTTP transaction is an object deﬁned for the lifetime of a single request from a client and the corresponding response from Trafﬁc Server. The ‘‘TSHttpTxn‘‘ structure is the main handle given to a plugin for manipulating a transaction’s internal state. Additionally, an HTTP transaction has a reference back to the HTTP session that created it. The sample code below illustrates how to register locally to a transaction and associate data to the transaction.
/* * Simple plugin that illustrates: * - how to register locally to a transaction * - how to deal with data that's associated with a tranaction * * Note: for readability, error checking is omitted */
#include 
310 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
#define DBG_TAG "txn"
/* Structure to be associated to txns */ typedef struct { int i; float f; char *s; } TxnData;
/* Allocate memory and init a TxnData structure */ TxnData * txn_data_alloc() { TxnData *data; data= TSmalloc( sizeof(TxnData));
 data->i=1; data->f= 0.5; data->s="Constant String"; return data; }
/* Free up a TxnData structure */ void txn_data_free(TxnData *data) { TSfree(data); }
/* Handler for event READ_REQUEST and TXN_CLOSE */ static int local_hook_handler (TSCont contp, TSEvent event, void *edata) { TSHttpTxn txnp= (TSHttpTxn) edata; TxnData *txn_data= TSContDataGet(contp); switch (event) { case TS_EVENT_HTTP_READ_REQUEST_HDR: /* Modify values of txn data */ txn_data->i=2; txn_data->f= 3.5; txn_data->s="Constant String 2"; break;
 case TS_EVENT_HTTP_TXN_CLOSE: /* Print txn data values */ TSDebug(DBG_TAG,"Txn data i=%d f=%f s=%s", txn_data->i, txn_data->f, txn_data->
˓→s);
/* Then destroy the txn cont and its data */ txn_data_free(txn_data); TSContDestroy(contp); break;
 default: TSAssert(!"Unexpected event"); break; }
4.7. Plugin Development 311 Apache-Trafﬁcserver-Server Documentation, latest
TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE); return 1; }
/* Handler for event TXN_START */ static int global_hook_handler (TSCont contp, TSEvent event, void *edata) { TSHttpTxn txnp= (TSHttpTxn) edata; TSCont txn_contp; TxnData *txn_data;
 switch (event) { case TS_EVENT_HTTP_TXN_START: /* Create a new continuation for this txn and associate data to it */ txn_contp= TSContCreate(local_hook_handler, TSMutexCreate()); txn_data= txn_data_alloc(); TSContDataSet(txn_contp, txn_data);
/* Registers locally to hook READ_REQUEST and TXN_CLOSE */ TSHttpTxnHookAdd(txnp, TS_HTTP_READ_REQUEST_HDR_HOOK, txn_contp); TSHttpTxnHookAdd(txnp, TS_HTTP_TXN_CLOSE_HOOK, txn_contp); break;
 default: TSAssert(!"Unexpected event"); break; }
TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE); return 1; } void TSPluginInit( int argc, const char *argv[]) { TSCont contp;
/* Note that we do not need a mutex for this txn since it registers globally and doesn't have any data associated with it */ contp= TSContCreate(global_hook_handler, NULL);
/* Register gloabally */ TSHttpHookAdd(TS_HTTP_TXN_START_HOOK, contp); }
See Adding Hooks for background on HTTP transactions and HTTP hooks, as well as Hooks and Transactions. See also the HTTP Transaction State Diagram for an illustration of the steps involved in a typical HTTP transaction. The HTTP transaction functions are: • TSHttpTxnCacheLookupStatusGet() • TSHttpTxnCachedReqGet() - Note that it is an error to modify cached headers. • TSHttpTxnCachedRespGet() - Note that it is an error to modify cached headers. • TSHttpTxnClientReqGet() - Plugins that must read client request headers use this call to retrieve the HTTP header.
312 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
• TSHttpTxnClientRespGet() • TSHttpTxnErrorBodySet() • TSHttpTxnHookAdd() • TSHttpTxnNextHopAddrGet() • TSHttpTxnParentProxySet() • TSHttpTxnReenable() • TSHttpTxnServerAddrGet() • TSHttpTxnServerAddrSet() • TSHttpTxnServerReqGet() • TSHttpTxnServerRespGet() • TSHttpTxnSsnGet() • TSHttpTxnTransformedRespCache() • TSHttpTxnTransformRespGet() • TSHttpTxnUntransformedRespCache()
Intercepting HTTP Transactions
The intercepting HTTP transaction functions enable plugins to intercept transactions either after the request is received or upon contact with the origin server. The plugin then acts as the origin server using the TSVConn interface. The intercepting HTTP transaction function allow for reading POST bodies in plugins as well as using alternative transports to the origin server.The intercepting HTTP transaction functions are: • TSHttpTxnIntercept()
Initiate HTTP Connection
This function enables plugins to initiate HTTP transactions. The initiate HTTP connection function is: • TSHttpConnect()
HTTP Alternate Selection
The HTTP alternate selection functions provide a mechanism for hooking into Trafﬁc Server’s alternate selection mechanism and augmenting it with additional information. HTTP alternate selection refers to the process of choos- ing between several alternate versions of a document for a speciﬁc URL. Alternates arise because the HTTP 1.1 speciﬁcation allows different documents to be sent back for the same URL (depending on the clients request). For example, a server might send back a GIF image to a client that only accepts GIF images, and might send back a JPEG image to a client that only accepts JPEG images. The alternate selection mechanism is invoked when Trafﬁc Server looks up a URL in its cache. For each URL, Trafﬁc Server stores a vector of alternates. For each alternate in this vector, Trafﬁc Server computes a quality value between 0 and 1 that represents how “good” the alternate is. A quality value of 0 means that the alternate is unacceptable; a value of 1 means that the alternate is a perfect match. If a plugin hooks onto the TS_HTTP_SELECT_ALT_HOOK, then it will be called back when Trafﬁc Server per- forms alternate selection. You cannot register locally to the hook TS_HTTP_SELECT_ALT_HOOK by using
4.7. Plugin Development 313 Apache-Trafﬁcserver-Server Documentation, latest
TSHttpTxnHookAdd - you can only do so by using only TSHttpHookAdd. Since Trafﬁc Server does not ac- tually have an HTTP transaction or an HTTP session on hand when alternate selection is performed, it is only valid to hook onto the global list of TS_HTTP_SELECT_ALT_HOOK. Trafﬁc Server calls each of the select alternate hooks with the TS_EVENT_HTTP_SELECT_ALT event. The void *edata argument that is passed to the continuation is a pointer to an TSHttpAltInfo structure. It can be used later to call the HTTP alternate selection functions listed at the end of this section. Unlike other hooks, this alternate selection callout is non-blocking; the expectation is that the quality value for the alternate will be changed by a call to TSHttpAltInfoQualitySet.
: HTTP SM does not have to be reenabled using TSHttpTxnReenable or any other APIs; just return from the function.
The sample code below shows how to call the alternate APIs. static void handle_select_alt(TSHttpAltInfo infop) { TSMBuffer client_req_buf, cache_resp_buf; TSMLoc client_req_hdr, cache_resp_hdr;
TSMLoc accept_transform_field; TSMLoc content_transform_field;
 int accept_transform_len=-1, content_transform_len=-1; const char* accept_transform_value= NULL; const char* content_transform_value= NULL; int content_plugin, accept_plugin;
 float quality;
/* get client request, cached request and cached response */ TSHttpAltInfoClientReqGet (infop,&client_req_buf,&client_req_hdr); TSHttpAltInfoCachedRespGet(infop,&cache_resp_buf,&cache_resp_hdr);
/* get the Accept-Transform field value from the client request */ accept_transform_field= TSMimeHdrFieldFind(client_req_buf, client_req_hdr,"Accept-Transform",-1); if (accept_transform_field) { TSMimeHdrFieldValueStringGet(client_req_buf, client_req_hdr, accept_transform_field,0,&accept_transform_value,&accept_transform_len); TSDebug(DBG_TAG,"Accept-Transform = |%s|", accept_transform_value); }
/* get the Content-Transform field value from cached server response */ content_transform_field= TSMimeHdrFieldFind(cache_resp_buf, cache_resp_hdr,"Content-Transform",-1); if (content_transform_field) { TSMimeHdrFieldValueStringGet(cache_resp_buf, cache_resp_hdr, content_transform_field,0,&content_transform_value,&content_transform_
˓→len); TSDebug(DBG_TAG,"Content-Transform = |%s|", content_transform_value); }
/* compute quality */ accept_plugin= (accept_transform_value&& (accept_transform_len>0)&& (strncmp(accept_transform_value,"plugin", accept_transform_len)==0));
314 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 content_plugin= (content_transform_value&& (content_transform_len>0)&& (strncmp(content_transform_value,"plugin", content_transform_len)==0));
 if (accept_plugin) { quality= content_plugin? 1.0: 0.0; } else { quality= content_plugin? 0.0: 0.5; }
TSDebug(DBG_TAG,"Setting quality to %3.1f", quality);
/* set quality for this alternate */ TSHttpAltInfoQualitySet(infop, quality);
/* cleanup */ if (accept_transform_field) TSHandleMLocRelease(client_req_buf, client_req_hdr, accept_transform_field); TSHandleMLocRelease(client_req_buf, TS_NULL_MLOC, client_req_hdr);
 if (content_transform_field) TSHandleMLocRelease(cache_resp_buf, cache_resp_hdr, content_transform_field); TSHandleMLocRelease(cache_resp_buf, TS_NULL_MLOC, cache_resp_hdr); } static int alt_plugin(TSCont contp, TSEvent event, void *edata) { TSHttpAltInfo infop;
 switch (event) { case TS_EVENT_HTTP_SELECT_ALT: infop= (TSHttpAltInfo)edata; handle_select_alt(infop); break;
 default: break; }
 return 0; } void TSPluginInit( int argc, const char *argv[]) { TSHttpHookAdd(TS_HTTP_SELECT_ALT_HOOK, TSContCreate (alt_plugin, NULL)); }
Trafﬁc Server augments the alternate selection through these callouts using the following algorithm: 1. Trafﬁc Server computes its own quality value for the alternate, taking into account the quality of the accept match, the encoding match, and the language match. 2. Trafﬁc Server then calls out each of the continuations on the global TS_HTTP_SELECT_ALT_HOOK‘s list. 3. It multiplies its quality value with the value returned by each callout. Since all of the values are clamped to be
4.7. Plugin Development 315 Apache-Trafﬁcserver-Server Documentation, latest
 between 0 and 1, the ﬁnal value will be between 0 and 1 as well. 4. This algorithm also ensures that a single callout can block the usage of a given alternate by specifying a quality value of 0. A common usage for the alternate selection mechanism is when a plugin transforms a document for some clients and not for others, but wants to store both the transformed and unchanged document. The client’s request will specify whether it accepted the transformed document. The plugin will then determine if the alternate matches this speciﬁca- tion and then set the appropriate quality level for the alternate. The HTTP alternate selection functions are: • TSHttpAltInfoCachedReqGet • TSHttpAltInfoCachedRespGet • TSHttpAltInfoClientReqGet • TSHttpAltInfoQualitySet
Hooks
To understand hooks and transactions, you should be familiar with the following terminology:
HTTP Transaction
A transaction consists of a single HTTP request from a client and the response Trafﬁc Server sends to that client. Thus, a transaction begins when Trafﬁc Server receives a request and ends when Trafﬁc Server sends the response. Trafﬁc Server uses HTTP state machines to process transactions. The state machines follow a complex set of states involved in sophisticated caching and document retrieval (taking into account, for example, alternate selection, fresh- ness criteria, and hierarchical caching). The Trafﬁc Server API provides hooks to a subset of these states, as illustrated in the HTTP Transaction State Diagram below.
Transform Hooks
The two transform hooks, TS_HTTP_REQUEST_TRANSFORM_HOOK and TS_HTTP_RESPONSE_TRANSFORM_HOOK, are called in the course of an HTTP transform. To see where in the HTTP transaction they are called, look for the “set up transform” ovals in the HTTP Transaction State Diagram below.
HTTP Session
A session consists of a single client connection to Trafﬁc Server; it may consist of a single transaction or several transactions in succession. The session starts when the client connection opens and ends when the connection closes.
HTTP Transaction State Diagram
HTTP Transacation Timers
For an overview of HTTP transaction timers, refer to the transaction timer diagram below.
316 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
HTTP Transaction Timers
Transaction Timers at various states
Trafﬁc Server runs a variety of timers at various states of a transaction. Typically, a given transaction may include upto two connections (one on the UA/client side and the other on the Origin side). Trafﬁc Server supports two kinds of timers “Active” and “Inactive” timers for each side respectively, as applicable at a given state. The below picture illustrates the speciﬁc timers run at various states in the current implementation. The picture only depicts the HTTP transaction level timers and does not include the TLS handshake states and other more detailed/internal timers in each individual state.
4.13: Transaction Timers in various states
Continuations
Activating Continuations
Continuations are activated when they receive an event or by TSContSchedule (which schedules a continuation to receive an event). Continuations might receive an event because: • Your plugin calls TSContCall • The Trafﬁc Server HTTP state machine sends an event corresponding to a particular HTTP hook • A Trafﬁc Server IO processor (such as a cache processor or net processor) is letting a continuation know there is data (cache or network) available to read or write. These callbacks are a result of using functions such TSVConnRead/Write or TSCacheRead/Write
Writing Handler Functions
The handler function is the key component of a continuation. It is supposed to examine the event and event data, and then do something appropriate. The probable action might be to schedule another event for the continuation to received, to open up a connection to a server, or simply to destroy itself. The continuation’s handler function is a function of type TSEventFunc. Its arguments are a continuation, an event, and a pointer to some data (this data is passed to the continuation by the caller - do not confuse this data with the continuation’s own data, associated by TSContDataSet()). When the continuation is called back, the continuation and an event are passed to the handler function. The continuation is a handle to the same continuation that is invoked. The handler function typically has a switch statement to handle the events it receives:
 static int some_handler (TScont contp, TSEvent event, void *edata) { // ..... switch(event) { case TS_EVENT_SOME_EVENT_1: do_some_thing_1; return; case TS_EVENT_SOME_EVENT_2: do_some_thing_2; return; case TS_EVENT_SOME_EVENT_3: do_some_thing_3; return;
4.7. Plugin Development 317 Apache-Trafﬁcserver-Server Documentation, latest
 default: break; } return 0; }
: You might notice that a continuation cannot determine if more events are “in ﬂight” toward it. Do not use TSContDestroy() to delete a continuation before you make sure that all incoming events, such as those sent because of TSHttpTxnHookAdd(), have been handled.
The following table lists events and the corresponding type of void* data passed to handler functions:
Event Event Sender Data Type TS_EVENT_HTTP_READ_REQUEST_HDR TS_HTTP_READ_REQUEST_HDR_HOOK TSHttpTxn TS_EVENT_HTTP_PRE_REMAP TS_HTTP_PRE_REMAP_HOOK TSHttpTxn TS_EVENT_HTTP_OS_DNS TS_HTTP_OS_DNS_HOOK TSHttpTxn TS_EVENT_HTTP_SEND_REQUEST_HDR TS_HTTP_SEND_REQUEST_HDR_HOOK TSHttpTxn TS_EVENT_HTTP_READ_CACHE_HDR TS_HTTP_READ_CACHE_HDR_HOOK TSHttpTxn TS_EVENT_HTTP_READ_RESPONSE_HDR TS_HTTP_READ_RESPONSE_HDR_HOOK TSHttpTxn TS_EVENT_HTTP_SEND_RESPONSE_HDR TS_HTTP_SEND_RESPONSE_HDR_HOOK TSHttpTxn TS_EVENT_HTTP_SELECT_ALT TS_HTTP_SELECT_ALT_HOOK TSHttpTxn TS_EVENT_HTTP_TXN_START TS_HTTP_TXN_START_HOOK TSHttpTxn TS_EVENT_HTTP_TXN_CLOSE TS_HTTP_TXN_CLOSE_HOOK TSHttpTxn TS_EVENT_HTTP_SSN_START TS_HTTP_SSN_START_HOOK TSHttpSsn TS_EVENT_HTTP_SSN_CLOSE TS_HTTP_SSN_CLOSE_HOOK TSHttpSsn TS_EVENT_NONE TS_EVENT_CACHE_LOOKUP_COMPLETE TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK TSHttpTxn TS_EVENT_IMMEDIATE TSVConnClose() TSVIOReenable() TSContSchedule() TS_EVENT_IMMEDIATE TS_HTTP_REQUEST_TRANSFORM_HOOK TS_EVENT_IMMEDIATE TS_HTTP_RESPONSE_TRANSFORM_HOOK TS_EVENT_CACHE_OPEN_READ TSCacheRead() Cache VC TS_EVENT_CACHE_OPEN_READ_FAILED TSCacheRead() TS_CACHE_ERROR code TS_EVENT_CACHE_OPEN_WRITE TSCacheWrite() Cache VC TS_EVENT_CACHE_OPEN_WRITE_FAILED TSCacheWrite() TS_CACHE_ERROR code TS_EVENT_CACHE_REMOVE TSCacheRemove() TS_EVENT_CACHE_REMOVE_FAILED TSCacheRemove() TS_CACHE_ERROR code TS_EVENT_NET_ACCEPT TSNetAccept() TSHttpTxnServerIntercept() TSHttpTxnIntercept() TSNetVConnection TS_EVENT_NET_ACCEPT_FAILED TSNetAccept() TSHttpTxnServerIntercept() TSHttpTxnIntercept() TS_EVENT_HOST_LOOKUP TSHostLookup() TSHostLookupResult TS_EVENT_TIMEOUT TSContSchedule() TS_EVENT_ERROR TS_EVENT_VCONN_READ_READY TSVConnRead() TSVIO TS_EVENT_VCONN_WRITE_READY TSVConnWrite() TSVIO TS_EVENT_VCONN_READ_COMPLETE TSVConnRead() TSVIO TS_EVENT_VCONN_WRITE_COMPLETE TSVConnWrite() TSVIO TS_EVENT_VCONN_EOS TSVConnRead() TSVIO TS_EVENT_NET_CONNECT TSNetConnect() TSVConn TS_EVENT_NET_CONNECT_FAILED TSNetConnect() TSVConn TS_EVENT_HTTP_CONTINUE TS_EVENT_HTTP_ERROR TS_EVENT_MGMT_UPDATE TSMgmtUpdateRegister()
318 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
The continuation functions are listed below: • TSContCall() • TSContCreate() • TSContDataGet() • TSContDataSet() • TSContDestroy() • TSContMutexGet() • TSContSchedule() The continuation interface is Trafﬁc Server’s basic callback mechanism. Continuations are instances of the opaque data type TSCont. In its basic form, a continuation represents a handler function and a mutex. This chapter covers the following topics: • Mutexes and Data • Activating Continuations • Writing Handler Functions
Mutexes and Data
A continuation must be created with a mutex if your continuation does one of the following: • is registered globally (TSHttpHookAdd or TSHttpSsnHookAdd) to an HTTP hook and uses TSContDataSet/Get • is registered locally (TSHttpTxnHookAdd), but for multiple transactions uses TSContDataSet/Get • uses TSCacheXXX, TSNetXXX, TSHostLookup, or TSContSchedule APIs Before being activated, a caller must grab the continuation’s mutex. This requirement makes it possible for a continua- tion’s handler function to safely access its data and to prevent multiple callers from running it at the same time (see the About the Sample Protocol for usage). The data protected by the mutex is any global or continuation data associated to the continuation by TSContDataSet. This does not include the local data created by the continuation handler function. A typical example of continuations created with associated data structures and mutexes is the transaction state machine created in the sample Protocol plugin (see One Way to Implement a Transaction State Machine). A reentrant call occurs when the continuation passed as an argument to the API can be called in the same stack trace as the function calling the API. For example, if you call TSCacheRead (contp, mykey), it is possible that contp‘s handler will be called directly and then TSCacheRead returns. Caveats that could cause issues include the following: • a continuation has data associated with it (TSContDataGet). • the reentrant call passes itself as a continuation to the reentrant API. In this case, the continuation should not try to access its data after calling the reentrant API. The reason for this is that data may be modiﬁed by the section of code in the continuation’s handler that handles the event sent by the API. It is recommended that you always return after a reentrant call to avoid accessing something that has been deallocated. Below is an example, followed by an explanation. continuation_handler (TSCont contp, TSEvent event, void *edata) { switch (event) { case event1: TSReentrantCall (contp);
4.7. Plugin Development 319 Apache-Trafﬁcserver-Server Documentation, latest
/* Return right away after this call */ break; case event2: TSContDestroy (contp); break; } }
The above example ﬁrst assumes that the continuation is called back with event1; it then does the ﬁrst reentrant call that schedules the continuation to receive event2. Because the call is reentrant, the processor calls back the continuation right away with event2 and the continuation is destroyed. If you try to access the continuation or one of its members after the reentrant call, then you might access something that has been deallocated. To avoid accessing something that has been deallocated, never access the continuation or any of its members after a reentrant call - just exit the handler. Note: Most HTTP transaction plugin continuations do not need non-null mutexes because they’re called within the processing of an HTTP transaction, and therefore have the transaction’s mutex. It is also possible to specify a continuation’s mutex as NULL. This should be done only when registering a continuation to a global hook, by a call to TSHttpHookAdd. In this case, the continuation can be called simultaneously by different instances of HTTP SM running on different threads. Having a mutex here would slow and/or hinder Trafﬁc Server performance, since all the threads will try to lock the same mutex. The drawback of not having a mutex is that such a continuation cannot have data associated with it (i.e., TSContDataGet/Set cannot be used). When using a NULL mutex it is dangerous to access the continuation’s data, but usually continuations with NULL mutexes have no data associated with them anyway. An example of such a continuation is one that gets called back every time an HTTP request is read, and then determines from the request alone if the request should go through or be rejected. An HTTP transaction gives its continuation data to the contp.
Mutexes
A mutex is the basic synchronization method used within Trafﬁc Server to protect data from simultaneous access by multiple threads. A mutex acts as a lock that protects data in one program thread from being accessed by another thread. The Trafﬁc Server API provides two functions that attempt to access and lock the data: TSMutexLockTry() and TSMutexLock(). TSMutexLock is a blocking call - if you use it, you can slow Trafﬁc Server performance because transaction processing pauses until the mutex is unlocked. It should be used only on threads created by the plugin TSContThreadCreate. Never use it on a continuation handler called back by the Cache, Net, or Event Processor. Even if the critical section is very small, do not use it. If you need to update a ﬂag, then set a variable and/or use atomic operations. If TSMutexLock() is used in any case other than the one recommended above, then the result will be a serious performance impact. TSMutexLockTry, on the other hand, attempts to lock the mutex only if it is unlocked (i.e., not being used by another thread). It should be used in all cases other than the above mentioned TSMutexLock case. If the TSMutexLockTry attempt fails, then you can schedule a future attempt (which must be at least 10 milliseconds later). In general, you should use TSMutexLockTry instead of TSMutexLock. • TSMutexLockTry is required if you are tying to lock Trafﬁc Server internal or system resources (such as the network, cache, event processor, HTTP state machines, and IO buffers). • TSMutexLockTry is required if you are making any blocking calls (such as network, cache, or ﬁle IO calls). • TSMutexLock might not be necessary if you are not making blocking calls and if you are only accessing local resources.
320 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
The Trafﬁc Server API uses the TSMutex type for a mutex. There are two typical uses of mutex. One use is for locking global data or data shared by various continuations. The other typical usage is for locking data associated with a continuation (i.e., data that might be accessed by other continuations).
Locking Global Data
The blacklist-1.c sample plugin implements a mutex that locks global data. The blacklist plugin reads its blacklisted sites from a conﬁguration ﬁle; ﬁle read operations are protected by a mutex created in TSPluginInit(). The blacklist-1.c code uses TSMutexLockTry() instead of TSMutexLock(). For more detailed information, see the blacklist-1.c code; start by looking at the TSPluginInit() function. General guidelines for locking shared data are as follows: 1. Create a mutex for the shared data with TSMutexCreate(). 2. Whenever you need to read or modify this data, ﬁrst lock it by calling TSMutexLockTry(); then read or modify the data. 3. When you are done with the data, unlock it with TSMutexUnlock(). If you are unlocking data accessed dur- ing the processing of an HTTP transaction, then you must unlock it before calling TSHttpTxnReenable().
Protecting a Continuation’s Data
You must create a mutex to protect a continuation’s data if it might be accessed by other continuations or processes. Here’s how: 1. Create a mutex for the continuation using TSMutexCreate. For example:
TSMutex mutexp; mutexp= TSMutexCreate ();
2. When you create the continuation, specify this mutex as the continuation’s mutex. For example:
TSCont contp; contp= TSContCreate (handler, mutexp);
If any other functions want to access contp‘s data, then it is up to them to get contp‘s mutex (using, for example, TSContMutexGet) to lock it. For usage, ssee the sample Protocol plugin.
How to Associate a Continuation With Every HTTP Transaction
There could be several reasons why you’d want to create a continuation for each HTTP transaction that calls back your plugin. Some potential scenarios are listed below. • You want to register hooks locally with the new continuation instead of registering them globally to the contin- uation plugin. • You want to store data speciﬁc to each HTTP transaction that you might need to reuse across various hooks. • You’re using APIs (like TSHostLookup) that will call back the continuation with a certain event.
4.7. Plugin Development 321 Apache-Trafﬁcserver-Server Documentation, latest
How to Add the New Continuation
A typical way of adding the new continuation is by registering the plugin continuation to be called back by HTTP transactions globally when they reach TS_HTTP_TXN_START_HOOK. Refer to the example below, which uses a transaction-speciﬁc continuation called txn_contp. void TSPluginInit(int argc, const char *argv[]) { /* Plugin continuation */ TSCont contp; if ((contp= TSContCreate (plugin_cont_handler, NULL))== TS_ERROR_PTR) { LOG_ERROR("TSContCreate"); } else { if (TSHttpHookAdd (TS_HTTP_TXN_START_HOOK, contp)== TS_ERROR) { LOG_ERROR("TSHttpHookAdd"); } } }
In the plugin continuation handler, create the new continuation txn_contp and then register it to be called back at TS_HTTP_TXN_CLOSE_HOOK: static int plugin_cont_handler(TSCont contp, TSEvent event, void *edata) { TSHttpTxn txnp= (TSHttpTxn)edata; TSCont txn_contp;
 switch (event) { case TS_EVENT_HTTP_TXN_START: /* Create the HTTP txn continuation */ txn_contp= TSContCreate(txn_cont_handler, NULL);
/* Register txn_contp to be called back when txnp reaches TS_HTTP_TXN_ ˓→CLOSE_HOOK */ if (TSHttpTxnHookAdd (txnp, TS_HTTP_TXN_CLOSE_HOOK, txn_contp)== TS_
˓→ERROR) { LOG_ERROR("TSHttpTxnHookAdd"); }
 break;
 default: TSAssert(!"Unexpected Event"); break; }
 if (TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE)== TS_ERROR) { LOG_ERROR("TSHttpTxnReenable"); }
 return 0; }
Remember that the txn_contp handler must destory itself when the HTTP transaction is closed. If you forget to do this, then your plugin will have a memory leak. static int txn_cont_handler(TSCont txn_contp, TSEvent event, void *edata) {
322 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSHttpTxn txnp; switch (event) { case TS_EVENT_HTTP_TXN_CLOSE: txnp= (TSHttpTxn) edata; TSContDestroy(txn_contp); break;
 default: TSAssert(!"Unexpected Event"); break; }
 if (TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE)== TS_ERROR) { LOG_ERROR("TSHttpTxnReenable"); }
 return 0; }
How to Store Data Speciﬁc to Each HTTP Transaction
For the example above, store the data in the txn_contp data structure - this means that you’ll create your own data structure. Now suppose you want to store the state of the HTTP transaction: typedef struct { int state; } ContData;
You need to allocate the memory and initialize this structure for each HTTP txnp. You can do that in the plugin continuation handler when it is called back with TS_EVENT_HTTP_TXN_START static int plugin_cont_handler(TSCont contp, TSEvent event, void *edata) { TSHttpTxn txnp= (TSHttpTxn)edata; TSCont txn_contp; ContData *contData;
 switch (event) { case TS_EVENT_HTTP_TXN_START: /* Create the HTTP txn continuation */ txn_contp= TSContCreate(txn_cont_handler, NULL);
/* Allocate and initialize the txn_contp data */ contData= (ContData *) TSmalloc(sizeof(ContData)); contData->state=0; if (TSContDataSet(txn_contp, contData)== TS_ERROR) { LOG_ERROR("TSContDataSet"); }
/* Register txn_contp to be called back when txnp reaches TS_HTTP_TXN_ ˓→CLOSE_HOOK */ if (TSHttpTxnHookAdd (txnp, TS_HTTP_TXN_CLOSE_HOOK, txn_contp)== TS_
˓→ERROR) { LOG_ERROR("TSHttpTxnHookAdd"); }
4.7. Plugin Development 323 Apache-Trafﬁcserver-Server Documentation, latest
 break;
 default: TSAssert(!"Unexpected Event"); break; }
 if (TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE)== TS_ERROR) { LOG_ERROR("TSHttpTxnReenable"); }
 return 0; }
For accessing this data from anywhere, use TSContDataGet:
TSCont txn_contp; ContData *contData; contData= TSContDataGet(txn_contp); if (contData== TS_ERROR_PTR) { LOG_ERROR("TSContDataGet"); } contData->state=1;
Remember to free this memory before destroying the continuation: static int txn_cont_handler(TSCont txn_contp, TSEvent event, void *edata) { TSHttpTxn txnp; ContData *contData; switch (event) { case TS_EVENT_HTTP_TXN_CLOSE: txnp= (TSHttpTxn) edata; contData= TSContDataGet(txn_contp); if (contData== TS_ERROR_PTR) { LOG_ERROR("TSContDataGet"); } else { TSfree(contData); } TSContDestroy(txn_contp); break;
 default: TSAssert(!"Unexpected Event"); break; }
 if (TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE)== TS_ERROR) { LOG_ERROR("TSHttpTxnReenable"); }
 return 0; }
324 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Using Locks
You do not need to use locks when a continuation has registered itself to be called back by HTTP hooks and it only uses the HTTP APIs. In the example above, the continuation txn_contp has registered itself to be called back at HTTP hooks and it only uses the HTTP APIs. In this case only, it’s safe to access data shared between txnp and txn_contp without grabbing a lock. In the example above, txn_contp is created with a NULL mutex. This works because the HTTP transaction txnp is the only one that will call back txn_contp, and you are guaranteed that txn_contp will be called back only one hook at a time. After processing is ﬁnished, txn_contp will reenable txnp. In all other cases, you should create a mutex with the continuation. In general, a lock is needed when you’re using iocore APIs or any other API where txn_contp is scheduled to be called back by a processor (such as the cache processor, the DNS processor, etc.). This ensures that txn_contp is called back sequentially and not simultaneously. In other words, you need to ensure that txn_contp will not be called back by both txnp and the cache processor at the same time, since this will result in a situation wherein you’re executing two pieces of code in conﬂict.
Special Case: Continuations Created for HTTP Transactions
If your plugin creates a new continuation for each HTTP transaction, then you probably don’t need to create a new mutex for it because each HTTP transaction (TSHttpTxn object) already has its own mutex. In the example below, it’s not necessary to specify a mutex for the continuation created in txn_handler:
 static void txn_handler (TSHttpTxn txnp, TSCont contp) { TSCont newCont; .... newCont= TSContCreate (newCont_handler, NULL); //It's not necessary to create a new mutex for newCont.
...
TSHttpTxnReenable (txnp, TS_EVENT_HTTP_CONTINUE); } static int test_plugin (TSCont contp, TSEvent event, void *edata) { TSHttpTxn txnp= (TSHttpTxn) edata;
 switch (event) { case TS_EVENT_HTTP_READ_REQUEST_HDR: txn_handler (txnp, contp); return 0; default: break; } return 0; }
The mutex functions are listed below: • TSMutexCreate() • TSMutexLock() • TSMutexLockTry()
4.7. Plugin Development 325 Apache-Trafﬁcserver-Server Documentation, latest
IO
This chapter contains the following sections:
Net Vconnections
A network vconnection (or netvconnection) is a wrapper around a TCP socket that enables the socket to work within the Trafﬁc Server vconnection framework. See vconnections for more information about the Trafﬁc Server abstraction for doing asynchronous IO. The netvconnection functions are listed below: • [dox ‘TSNetAccept’] in [dox “TSNetAccept” :src_ﬁle] • [dox %TSNetConnect%] in [dox :src_ﬁle]
Transformations
Vconnection Implementor’s View
A VConnection implementor writes only transformations. All other VConnections (net VConnections and cache VConnections) are implemented in iocore. As mentioned earlier, a given vconnection can have a maximum of one read operation and one write operation being performed on it. The vconnection user gets information about the operation being performed by examining the VIO returned by a call to TSVConnRead or TSVConnWrite. The implementor, in turn, gets a handle on the VIO operation by examining the VIO returned by TSVConnReadVIOGet or TSVConnWriteVIOGet (recall that every vconnection created through the Trafﬁc Server API has an associated read VIO and write VIO, even if it only supports reading or writing). For example, the null transform plugin’s transformation examines the input VIO by calling:
 input_vio= TSVConnWriteVIOGet (contp);
 where contp is the transformation. A vconnection is a continuation. This means it has a handler function that is run when an event is sent to it, or more accurately, when an event that was sent to it is received. It is the handler function’s job to examine the event, the current state of its read VIO and write VIO, and any other internal state the vconnection might have and potentially make some progress on the IO operations. It is common for the handler function for all vconnections to look similar. Their basic form looks something like the code fragment below:
 int vconnection_handler (TSCont contp, TSEvent event, void *edata) { if (TSVConnClosedGet (contp)) { /* Destroy any vconnection specific data here. */ TSContDestroy (contp); return 0; } else { /* Handle the incoming event */ } }
This code fragment basically shows that many vconnections simply want to destroy themselves when they are closed. However, the situation might also require the vconnection to do some cleanup processing - which is why TSVConnClose does not simply just destroy the vconnection.
326 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Vconnections are state machines that are animated by the events they receive. An event is sent to the vconnection whenever an TSVConnRead, TSVConnWrite, TSVConnClose, TSVConnShutdown, or TSVIOReenable call is performed. TSVIOReenable indirectly references the vconnection through a back-pointer in the VIO struc- ture to the vconnection. The vconnection itself only knows which call was performed by examining its state and the state of its VIOs. For example, when TSVConnClose is called, the vconnection is sent an immediate event (TS_EVENT_IMMEDIATE). For every event the vconnection receives, it needs to check its closed ﬂag to see if it has been closed. Similarly, when TSVIOReenable is called, the vconnection is sent an immediate event. For every event the vconnection receives, it must check its VIOs to see if the buffers have been modiﬁed to a state in which it can continue processing one of its operations. Finally, a vconnection is likely the user of other vconnections. It also receives events as the user of these other vconnections. When it receives such an event, like TS_EVENT_VCONN_WRITE_READY, it might just enable another vconnection that’s writing into the buffer used by the vconnection reading from it. The above description is merely intended to give the overall idea for what a vconnection needs to do.
Transformation VConnection
A transformation is a speciﬁc type of vconnection. It supports a subset of the vconnection functionality that enables one or more transformations to be chained together. A transformation sits as a bottleneck between an input data source and an output data sink, which enables it to view and modify all the data passing through it. Alternatively, some transformations simply scan the data and pass it on. A common transformation is one that compresses data in some manner. A transformation can modify either the data stream being sent to an HTTP client (e.g. the document) or the data stream being sent from an HTTP client (e.g. post data). To do this, the transformation should hook on to one of the following hooks: • TS_HTTP_REQUEST_TRANSFORM_HOOK • TS_HTTP_RESPONSE_TRANSFORM_HOOK Note that because a transformation is intimately associated with a given transaction, it is only possible to add the hook to the transaction hooks - not to the global or session hooks. Transformations reside in a chain, so their ordering is quite easily determined: transformations that add themselves to the chain are simply appended to it. Data is passed in to the transformation by initiating a vconnection write operation on the transformation. As a conse- quence of this design, a transformation must support the vconnection write operation. In other words, your transforma- tion must expect an upstream vconnection to write data to it. The transformation has to read the data, consume it, and tell the upstream vconnection it is ﬁnished by sending it an TS_EVENT_WRITE_COMPLETE event. Transformations cannot send the TS_EVENT_VCONN_WRITE_COMPLETE event to the upstream vconnection unless they are ﬁnished consuming all incoming data. If TS_EVENT_VCONN_WRITE_COMPLETE is sent prematurely, then certain internal Trafﬁc Server data structures will not be deallocated, thereby causing a memory leak. Here’s how to make sure that all incoming data is consumed: • After reading or copying data, make sure that you consume the data and increase the value of ndone for the input VIO, as in the following example taken from null-transform.c:
TSIOBufferCopy (TSVIOBufferGet (data->output_vio), TSVIOReaderGet (input_vio), towrite,0); /* Tell the read buffer that we have read the data and are no longer interested ˓→in it. */ TSIOBufferReaderConsume (TSVIOReaderGet (input_vio), towrite); /* Modify the input VIO to reflect how much has been read.*/ TSVIONDoneSet (input_vio, TSVIONDoneGet (input_vio)+ towrite);
• Before sending TS_EVENT_VCONN_WRITE_COMPLETE, your transformation should check the number of bytes remaining in the upstream vconnection’s write VIO (input VIO) using the function TSVIONTodoGet
4.7. Plugin Development 327 Apache-Trafﬁcserver-Server Documentation, latest
(input_vio). This value should go to zero when all of the upstream data is consumed (TSVIONTodoGet = nbytes - ndone). Do not send TS_EVENT_VCONN_WRITE_COMPLETE events if TSVIONTodoGet is greater than zero. • The transformation passes data out of itself by using the output vconnection retrieved by TSTransformOutputVConnGet. Immediately before Trafﬁc Server initiates the write operation (which inputs data into the transformation), it sets the output vconnection either to the next transformation in the chain of transformations or to a special terminating transformation (if it’s the last transformation in the chain). Since the transformation is handed ownership of the output vconnection, it must close it at some point in order for it to be deallocated. • All of the transformations in a transformation chain share the transaction’s mutex. This small restriction (en- forced by TSTransformCreate) removes many of the locking complications of implementing general vcon- nections. For example, a transformation does not have to grab its write VIO mutex before accessing its write VIO because it knows it already holds the mutex. The transformation functions are: * TSTransformCreate() * TSTransformOutputVConnGet()
VIOs
A VIO, or virtual IO, is a description of an IO operation that’s currently in progress. The VIO data structure is used by vconnection users to determine how much progress has been made on a particular IO operation and to re-enable an IO operation when it stalls due to buffer space issues. VIOs are used by vconnection implementors to determine the buffer for an IO operation, how much work to do on the IO operation, and which continuation to call back when progress on the IO operation is made. The TSVIO data structure itself is opaque, but it could be deﬁned as follows: typedef struct { TSCont continuation; TSVConn vconnection; TSIOBufferReader reader; TSMutex mutex; int nbytes; int ndone; } *TSVIO;
The VIO functions below access and modify various parts of the data structure. • TSVIOBufferGet() • TSVIOVConnGet() • TSVIOContGet() • TSVIOMutexGet() • TSVIONBytesGet() • TSVIONBytesSet() • TSVIONDoneGet() • TSVIONDoneSet() • TSVIONTodoGet() • TSVIOReaderGet() • TSVIOReenable()
328 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
IO Buffers
The IO buffer data structure is the building block of the vconnection abstraction. An IO buffer (TSIOBuffer) is composed of a list of buffer blocks that point to buffer data. Both the buffer block (TSIOBufferBlock) and buffer data (TSIOBufferData) data structures are reference-counted, so they can reside in multiple buffers at the same time. This makes it extremely efﬁcient to copy data from one IO buffer to another via TSIOBufferCopy, since Trafﬁc Server must only copy pointers and adjust reference counts appropriately (and doesn’t actually copy any data). The IO buffer abstraction provides for a single writer and multiple readers. In order for the readers to have no knowl- edge of each other, they manipulate IO buffers through the TSIOBufferReader data structure. Since only a single writer is allowed, there is no corresponding TSIOBufferWriter data structure. The writer simply modiﬁes the IO buffer directly. To see an example that illustrates how to use IOBuffers, refer to the sample code in the description of TSIOBufferBlockReadStart(). Additional information about IO buffer functions: • The TSIOBufferReader data structure tracks how much data in TSIOBuffer has been read. It has an offset number of bytes that is the current start point of a particular buffer reader (for every read operation on an TSIOBuffer, you must allocate an TSIOBufferReader). • Bytes that have already been read may not necessarily be freed within the TSIOBuffer. To consume bytes that have been read, you must call TSIOBufferConsume.
Cache API
The cache API enables plugins to read, write, and remove objects in the Trafﬁc Server cache. All cache APIs are keyed by an object called an TSCacheKey; cache keys are created via TSCacheKeyCreate; keys are destroyed via TSCacheKeyDestroy. Use TSCacheKeyDigestSet to set the hash of the cache key. Note that the cache APIs differentiate between HTTP data and plugin data. The cache APIs do not allow you to write HTTP docs in the cache; you can only write plugin-speciﬁc data (a speciﬁc type of data that differs from the HTTP type). Example: const unsigned char *key_name="example key name";
TSCacheKey key; TSCacheKeyCreate (&key); TSCacheKeyDigestSet (key, (unsigned char *) key_name , strlen(key_name)); TSCacheKeyDestroy (key);
Cache Reads
TSCacheRead does not really read - it is used for lookups (see the sample Protocol plugin). Possible callback events include: • TS_EVENT_CACHE_OPEN_READ - indicates the lookup was successful. The data passed back along with this event is a cache vconnection that can be used to initiate a read on this keyed data. • TS_EVENT_CACHE_OPEN_READ_FAILED - indicates the lookup was unsuccessful. Reasons for this event could be that another continuation is writing to that cache location, or the cache key doesn’t refer to a cached resource. Data payload for this event indicates the possible reason the read failed (TSCacheError).
4.7. Plugin Development 329 Apache-Trafﬁcserver-Server Documentation, latest
Cache Writes
Use TSCacheWrite to write to a cache (see the sample Protocol plugin). Possible callback events include: • TS_EVENT_CACHE_WRITE_READ - indicates the lookup was successful. The data passed back along with this event is a cache vconnection that can be used to initiate a cache write. • TS_EVENT_CACHE_OPEN_WRITE_FAILED - event returned when another continuation is currently writing to this location in the cache. Data payload for this event indicates the possible reason for the write failing (TSCacheError).
Cache Removes
Use TSCacheRemove to remove items from the cache. Possible callback events include: • TS_EVENT_CACHE_REMOVE - the item was removed. There is no data payload for this event. • TS_EVENT_CACHE_REMOVE_FAILED - indicates the cache was unable to remove the item identiﬁed by the cache key. TSCacheError data indicates why the remove failed.
Errors
Errors pertaining to the failure of various cache operations are indicated by TSCacheError (enumeration). They are as follows: • TS_CACHE_ERROR_NO_DOC - the key does not match a cached resource • TS_CACHE_ERROR_DOC_BUSY - e.g, another continuation could be writing to the cache location • TS_CACHE_ERROR_NOT_READY - the cache is not ready
Example
In the example below, suppose there is a cache hit and the cache returns a vconnection that enables you to read the document from cache. To do this, you need to prepare a buffer (cache_bufp) to hold the document; meanwhile, use TSVConnCachedObjectSizeGet to ﬁnd out the actual size of the document (content_length). Then, issue TSVConnRead to read the document with the total data length required as content_length. Assume the following data:
TSIOBuffer cache_bufp= TSIOBufferCreate (); TSIOBufferReader cache_readerp= TSIOBufferReaderAlloc (out_bufp); TSVConn cache_vconnp= NULL; TSVIO cache_vio= NULL; int content_length=0;
In the TS_CACHE_OPEN_READ handler: cache_vconnp= (TSVConn) data; TSVConnCachedObjectSizeGet (cache_vconnp,&content_length); cache_vio= TSVConnRead (cache_vconn, contp, cache_bufp, content_length);
In the TS_EVENT_VCONN_READ_READY handler:
330 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
(usual VCONN_READ_READY handler logic) int nbytes= TSVIONBytesGet (cache_vio); int ntodo= TSVIONTodoGet (cache_vio); int ndone= TSVIONDoneGet (cache_vio); (consume data in cache_bufp) TSVIOReenable (cache_vio);
Do not try to get continuations or VIOs from TSVConn objects for cache vconnections. Also note that the following APIs can only be used on transformation vconnections and must not be used on cache vconnections or net vconnec- tions: • TSVConnWriteVIOGet • TSVConnReadVIOGet • TSVConnClosedGet APIs such as TSVConnRead, TSVConnWrite, TSVConnClose, TSVConnAbort, and TSVConnShutdown can be used on any kind of vconnections. When you are ﬁnished: TSCacheKeyDestroy (key);
Vconnections
A User’s Perspective
To use a vconnection, a user must ﬁrst get a handle to one. This is usually accomplished by having it handed to the user; the user may also simply issue a call that creates a vconnection (such as TSNetConnect). In the case of transform plugins, the plugin creates a transformation vconnection viav TSTransformCreate and then accesses the output vconnection using TSTransformOutputVConnGet. After getting a handle to a vconnection, the user can then issue a read or write call. It’s important to note that not all vconnections support both reading and writing - as of yet, there has not been a need to query a vconnection about whether it can perform a read or write operation. That ability should be obvious from context. To issue a read or write operation, a user calls TSVConnRead or TSVConnWrite. These two operations both return VIO (TSVIO). The VIO describes the operation being performed and how much progress has been made. Transform plugins initiate output to the downstream vconnection by calling TSVConnWrite. A vconnection read or write operation is different from a normal UNIX read(2) or write(2) operation. Specif- ically, the vconnection operation can specify more data to be read or written than exists in the buffer handed to the operation. For example, it’s typical to issue a read for INT64_MAX (9 quintillion) bytes from a network vconnection in order to read all the data from the network connection until the end of stream is reached. This contrasts with the usual UNIX fashion of issuing repeated calls to read(2) until one of the calls ﬁnally returns 0 to indicate the end of stream was reached (indeed, the underlying implementation of vconnections on UNIX still does issue those calls to read(2), but the interface does not expose that detail). At most, a given vconnection can have one read operation and one write operation being performed on it. This is restricted both by design and common sense: if two write operations were performed on a single vconnection, then the user would not be able to specify which should occur ﬁrst and the output would occur in an intermingled fashion. Note that both a read operation and a write operation can happen on a single vconnection at the same time; the restriction is for more than one operation of the same type. One obvious issue is that the buffer passed to TSVConnRead and TSVConnWrite won’t be large enough - there is no reasonable way to make a buffer that can hold INT64_MAX (9 quintillion) bytes! The secret is that vcon- nections engage in a protocol whereby they signal to the user (via the continuation passed to TSVConnRead and
4.7. Plugin Development 331 Apache-Trafﬁcserver-Server Documentation, latest
TSVConnWrite) that they have emptied the buffers passed to them and are ready for more data. When this occurs, it is up to the user to add more data to the buffers (or wait for more data to be added) and then wake up the vconnection by calling TSVIOReenable on the VIO describing the operation. TSVIOReenable speciﬁes that the buffer for the operation has been modiﬁed and that the vconnection should reexamine it to see if it can make further progress. The null transform plugin provides an example of how this is done. Below is a prototype for TSVConnWrite:
TSVIO TSVConnWrite (TSVConn connp, TSCont contp, TSIOBufferReader readerp, int nbytes)
The connp is the vconnection the user is writing to and contp is the “user” - i.e., the continuation that connp calls back when it has emptied its buffer and is ready for more data. The call made in the null transform plugin is:
TSVConnWrite (output_conn, contp, data->output_reader, TSVIONBytesGet (input_vio));
In the example above, contp is the transformation vconnection that is writing to the output vconnection. The number of bytes to be written is obtained from input_vio by TSVIONBytesGet. When a vconnection calls back its user to indicate that it wants more data (or when some other condition has occurred), it issues a call to TSContCall. It passes the TSVIO describing the operation as the data parameter, and one of the values below as the event parameter. TS_EVENT_ERROR Indicates an error has occurred on the vconnection. This will happen for network IO if the underlying read(2) or write(2) call returns an error. TS_EVENT_VCONN_READ_READY The vconnection has placed data in the buffer passed to an TSVConnRead operation and it would like to do more IO, but the buffer is now full. When the user consumes the data from the buffer, this should re-enable the VIO so it indicates to the vconnection that the buffer has been modiﬁed. TS_EVENT_VCONN_WRITE_READY The vconnection has removed data from the buffer passed to an TSVConnWrite operation and it would like to do more IO, but the buffer does not have enough data in it. When placing more data in the buffer, the user should re-enable the VIO so it indicates to the vconnection that the buffer has been modiﬁed. TS_EVENT_VCONN_READ_COMPLETE The vconnection has read all the bytes speciﬁed by an TSVConnRead operation. The vconnection can now be used to initiate a new IO operation. TS_EVENT_VCONN_WRITE_COMPLETE The vconnection has written all the bytes speciﬁed by an TSVConnWrite operation and can now be used to initiate a new IO operation. TS_EVENT_VCONN_EOS An attempt was made to read past the end of the stream of bytes during the handling of an TSVConnRead operation. This event occurs when the number of bytes available for reading from a vconnection is less than the number of bytes the user speciﬁes should be read from the vconnection in a call to TSVConnRead. A common case where this occurs is when the user speciﬁes that INT64_MAX bytes are to be read from a network connection. For example: the null transform plugin’s transformation receives TS_EVENT_VCONN_WRITE_READY and TS_EVENT_VCONN_WRITE_COMPLETE events from the downstream vconnection as a result of the call to TSVConnWrite. After using a vconnection, the user must call TSVConnClose or TSVConnAbort. While both calls indicate that the vconnection can destroy itself, TSVConnAbort should be used when the connection is being closed abnormally. After a call to TSVConnClose or TSVConnAbort, the user will not be called back by the vconnection again. Sometimes it’s desirable to simply close down the write portion of a connection while keeping the read portion open. This can be accomplished via the TSVConnShutdown function, which shuts down either the read or write portion of a vconnection. Shutdown means that the vconnection will no longer call back the user with events for the portion of the connection that was shut down. For example: if the user shuts down the write portion of a connection, then the TS_EVENT_VCONN_WRITE_READY or TS_EVENT_VCONN_WRITE_COMPLETE events will not be produced.
332 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
In the null transform plugin, the write operation is shut down with a call to TSVConnShutdown. To learn how vconnections are used in transformation plugins, see Writing Content Transform Plugins. The vconnection functions are listed below: • TSVConnAbort() • TSVConnClose() • TSVConnClosedGet() • TSVConnRead() • TSVConnReadVIOGet() • TSVConnShutdown() • TSVConnWrite() • TSVConnWriteVIOGet()
HTTP Headers
Trafﬁc Server HTTP Header System
No Null-Terminated Strings
It’s not safe to assume that string data contained in marshal buffers (such as URLs and MIME ﬁelds) is stored in null-terminated string copies. Therefore, your plugins should always use the length parameter when retrieving or manipulating these strings. You cannot pass in NULL for string-length return values; string values returned from marshall buffers are not null-terminated. If you need a null-terminated value, then use TSstrndup to automatically null-terminate a string. The strings that come back and are not null-terminated cannot be passed into the common str*() routines
: Values returned from a marshall buffer can be NULL, which means the ﬁeld or object requested does not exist.
For example (from the blacklist-1 sample)
 char *host_string; int host_length; host_string= TSUrlHostGet (bufp, url_loc,&host_length); for (i=0; i< nsites; i++){ if (strncmp (host_string, sites[i], host_length)==0){ // ... }
See the sample plugins for additional examples.
Duplicate MIME Fields Are Not Coalesced
MIME headers can contain more than one MIME ﬁeld with the same name. Earlier versions of Trafﬁc Server joined multiple ﬁelds with the same name into one ﬁeld with composite values. This behavior came at a performance cost and caused interoperability problems with older clients and servers. Therefore, this version of Trafﬁc Server does not coalesce duplicate ﬁelds.
4.7. Plugin Development 333 Apache-Trafﬁcserver-Server Documentation, latest
Properly-behaving plugins should check for the presence of duplicate ﬁelds and then iterate over the duplicate ﬁelds via TSMimeHdrFieldNextDup().
MIME Fields Always Belong to an Associated MIME Header
When using Trafﬁc Server, you cannot create a new MIME ﬁeld without an associated MIME header or HTTP header; MIME ﬁelds are always seen as part of a MIME header or HTTP header. To use a MIME ﬁeld, you must specify the MIME header or HTTP header to which it belongs - this is called the ﬁeld’s parent header. The TSMimeField* functions in older versions of the SDK have been deprecated, as they do not require the parent header as inputs. The current version of Trafﬁc Server uses new functions, the ‘‘TSMimeHdrField‘‘ series, which require you to specify the location of the parent header along with the location of the MIME ﬁeld. For every deprecated ‘‘TSMimeField‘‘ function, there is a new, preferred TSMimeHdrField* function. Therefore, you should use the ‘‘TSMimeHdrField‘‘ functions instead of the deprecated ‘‘TSMimeField‘‘ series. Examples are provided below. Instead of:
TSMLoc TSMimeFieldCreate (TSMBuffer bufp)
You should use:
TSMLoc TSMimeHdrFieldCreate (TSMBuffer bufp, TSMLoc hdr)
Instead of: void TSMimeFieldCopyValues (TSMBuffer dest_bufp, TSMLoc dest_offset, TSMBuffer src_bufp, TSMLoc src_offset)
You should use: void TSMimeHdrFieldCopyValues (TSMBuffer dest_bufp, TSMLoc dest_hdr, TSMLoc dest_field, TSMBuffer src_bufp, TSMLoc src_hdr, TSMLoc src_field)
In the TSMimeHdrField* function prototypes, the TSMLoc ﬁeld corresponds to the TSMLoc offset used the dep- recated TSMimeField* functions (see the discussion of parent TSMLoc in the following section).
Release Marshal Buffer Handles
When you fetch a component object or create a new object, you get back a handle to the object location. The handle is either an TSMLoc for an object location or char * for a string location. You can manipulate the object through these handles, but when you are ﬁnished you need to release the handle to free up system resources. The general guideline is to release all TSMLoc and string handles you retrieve. The one exception is the string returned by TSUrlStringGet, which must be freed by a call to TSfree. The handle release functions expect three arguments: the marshal buffer containing the data, the location of the parent object, and the location of the object to be released. The parent location is usually clear from the creation of the TSMLoc or string. For example, if your plugin had the following calls: url_loc= TSHttpHdrUrlGet (bufp, hdr_loc); host_string= TSUrlHostGet (bufp, url_loc,&host_length); then your plugin would have to call:
334 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSHandleMLocRelease (bufp, hdr_loc, url_loc);
If an TSMLoc is obtained from a transaction, then it does not have a parent TSMLoc. Use the null TSMLoc constant TS_NULL_MLOC as its parent. For example, if your plugin calls:
TSHttpTxnClientReqGet (txnp,&bufp,&hdr_loc);
 then you must release hdr_loc with:
TSHandleMLocRelease (bufp, TS_NULL_MLOC, hdr_loc);
You need to use TS_NULL_MLOC to release any TSMLoc handles retrieved by the TSHttpTxn*Get functions. Here’s an example using a new TSMimeHdrField function:
TSHttpTxnServerRespGet( txnp,&resp_bufp,&resp_hdr_loc ); new_field_loc= TSMimeHdrFieldCreate (resp_bufp, resp_hdr_loc); TSHandleMLocRelease ( resp_bufp, resp_hdr_loc, new_field_loc); TSHandleMLocRelease ( resp_bufp, TS_NULL_MLOC, resp_hdr_loc);
See the sample plugins for many more examples.
: You should release handles before reenabling the HTTP transaction. In other words, call TSHandleMLocRelease before TSHttpTxnReenable.
Header Functions
The Trafﬁc Server API HTTP header functions enable you to work with HTTP header data stored in marshal buffers. The HTTP header data structure is a parsed version of the HTTP header deﬁned in the HTTP protocol speciﬁcation. An HTTP header is composed of a request or response line followed by zero or more MIME ﬁelds. In fact, an HTTP header is a subclass of a MIME header; all of the MIME header routines operate on HTTP headers. An HTTP request line is composed of a method, a URL, and version. A response line is composed of a version, status code, and reason phrase. See About HTTP Headers for additional details and examples. To facilitate fast comparisons and reduce storage size, Trafﬁc Server deﬁnes several pre-allocated method names. These names correspond to the methods deﬁned in the HTTP 1.1 speciﬁcation TS_HTTP_METHOD_CONNECT “CONNECT” TS_HTTP_METHOD_DELETE “DELETE” TS_HTTP_METHOD_GE “GET” TS_HTTP_METHOD_HEAD “HEAD” TS_HTTP_METHOD_ICP_QUERY “ICP_QUERY” TS_HTTP_METHOD_OPTIONS “OPTIONS” TS_HTTP_METHOD_POST “POST” TS_HTTP_METHOD_PURGE “PURGE” TS_HTTP_METHOD_PUT “PUT” TS_HTTP_METHOD_TRACE “TRACE” TS_HTTP_METHOD_PUSH “PUSH”
4.7. Plugin Development 335 Apache-Trafﬁcserver-Server Documentation, latest
Trafﬁc Server also deﬁnes several common values that appear in HTTP headers. TS_HTTP_VALUE_BYTES “bytes” TS_HTTP_VALUE_CHUNKED “chunked” TS_HTTP_VALUE_CLOSE “close” TS_HTTP_VALUE_COMPRESS “compress” TS_HTTP_VALUE_DEFLATE “deﬂate” TS_HTTP_VALUE_GZIP “gzip” TS_HTTP_VALUE_IDENTITY “identity” TS_HTTP_VALUE_KEEP_ALIVE “keep-alive” TS_HTTP_VALUE_MAX_AGE “max-age” TS_HTTP_VALUE_MAX_STALE “max-stale” TS_HTTP_VALUE_MIN_FRESH “min-fresh” TS_HTTP_VALUE_MUST_REVALIDATE “must-revalidate” TS_HTTP_VALUE_NONE “none” TS_HTTP_VALUE_NO_CACHE “no-cache” TS_HTTP_VALUE_NO_STORE “no-store” TS_HTTP_VALUE_NO_TRANSFORM “no-transform” TS_HTTP_VALUE_ONLY_IF_CACHED “only-if-cached” TS_HTTP_VALUE_PRIVATE “private” TS_HTTP_VALUE_PROXY_REVALIDATE “proxy-revalidate” TS_HTTP_VALUE_PUBLIC “public” TS_HTTP_VALUE_S_MAX_AGE “s-maxage” The method names and header values above are deﬁned in ts.h as const char* strings. When Trafﬁc Server sets a method or a header value, it checks to make sure that the new value is one of the known values. If it is, then it stores a pointer into a global table (instead of storing the known value in the marshal buffer). The method names and header values listed above are also pointers into this table. This allows simple pointer comparison of the value returned from TSHttpMethodGet with one of the values listed above. It is also recommended that you use the above values when referring to one of the known schemes, since this removes the possibility of a spelling error. The HTTP Header Functions are listed below: • TSHttpHdrClone() • TSHttpHdrCopy() • TSHttpHdrCreate() • TSHttpHdrDestroy() • TSHttpHdrLengthGet() • TSHttpHdrMethodGet() • TSHttpHdrMethodSet() • TSHttpHdrPrint() • TSHttpHdrReasonGet()
336 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
• TSHttpHdrReasonLookup() • TSHttpHdrReasonSet() • TSHttpHdrStatusGet() • TSHttpHdrStatusSet() • TSHttpHdrTypeGet() • TSHttpHdrTypeSet() • TSHttpHdrUrlGet() • TSHttpHdrUrlSet() • TSHttpHdrVersionGet() • TSHttpHdrVersionSet() • TSHttpParserClear() • TSHttpParserCreate() • TSHttpParserDestroy() • TSHttpHdrParseReq() • TSHttpHdrParseResp()
MIME Headers
The Trafﬁc Server **MIME header functions ** enable you to retrieve and modify information about HTTP MIME ﬁelds. An HTTP request or response consists of a header, body, and trailer. The HTTP header contains a request (or response) line and a MIME header. A MIME header is composed of zero or more MIME ﬁelds. A MIME ﬁeld is composed of a ﬁeld name, a colon, and zero or more ﬁeld values (values in a ﬁeld are separated by commas). In the example below: Foo is the MIME ﬁeld name, bar is the ﬁrst MIME ﬁeld value, and car is the second MIME ﬁeld value.
Foo: bar, car
The following example is an augmented Backus-Naur Form (BNF) for the form of a MIME header - it speciﬁes exactly what was described above. A header consists of zero or more ﬁelds that contain a name, separating colon, and zero or more values. A name or value is simply a string of tokens that is potentially zero length; a token is any character except certain control characters and separators (such as colons). For the purpose of retrieving a ﬁeld, ﬁeld names are not case-sensitive; therefore, the ﬁeld names Foo, foo and fOO are all equivalent.
MIME-header= *MIME-field MIME-field= field-name":" #field-value field-name= *token field-value= *token
The MIME header data structure is a parsed version of a standard Internet MIME header. The MIME header data structure is similar to the URL data structure (see URLs). The actual data is stored in a marshal buffer; the MIME header functions operate on a marshal buffer and a location (TSMLoc) within the buffer. After a call to TSMimeHdrFieldDestroy, TSMimeHdrFieldRemove, or TSUrlDestroy is made, you must deallocate the TSMLoc handle with a call to TSHandleMLocRelease. You do not need to deallocate a NULL handles. For example: if you call TSMimeHdrFieldValueStringGet to get the value of the content type ﬁeld
4.7. Plugin Development 337 Apache-Trafﬁcserver-Server Documentation, latest
 and the ﬁeld does not exist, then it returns TS_NULL_MLOC. In such a case, you wouldn’t need to deallocate the handle with a call to TSHandleMLocRelease. The location (TSMLoc) in the MIME header functions can be either an HTTP header location or a MIME header location. If an HTTP header location is passed to these functions, then the system locates the MIME header associated with that HTTP header and executes the corresponding MIME header operations speciﬁed by the functions (see the example in the description of TSMimeHdrCopy()). Note: MIME headers may contain more than one MIME ﬁeld with the same name. Previous versions of Trafﬁc Server joined multiple ﬁelds with the same name into one ﬁeld with composite values, but this behavior came at a performance cost and caused compatability issues with older clients and servers. Hence, the current version of Trafﬁc Server does not coalesce duplicate ﬁelds. Correctly-behaving plugins should check for the presence of duplicate ﬁelds and iterate over the duplicate ﬁelds by using TSMimeHdrFieldNextDup. To facilitate fast comparisons and reduce storage size, Trafﬁc Server deﬁnes several pre-allocated ﬁeld names. These ﬁeld names correspond to the ﬁeld names in HTTP and NNTP headers. TS_MIME_FIELD_ACCEPT “Accept” TS_MIME_LEN_ACCEPT TS_MIME_FIELD_ACCEPT_CHARSET “Accept-Charset” TS_MIME_LEN_ACCEPT_CHARSET TS_MIME_FIELD_ACCEPT_ENCODING “Accept-Encoding” TS_MIME_LEN_ACCEPT_ENCODING TS_MIME_FIELD_ACCEPT_LANGUAGE “Accept-Language” TS_MIME_LEN_ACCEPT_LANGUAGE TS_MIME_FIELD_ACCEPT_RANGES “Accept-Ranges” TS_MIME_LEN_ACCEPT_RANGES TS_MIME_FIELD_AGE “Age” TS_MIME_LEN_AGE TS_MIME_FIELD_ALLOW “Allow” TS_MIME_LEN_ALLOW TS_MIME_FIELD_APPROVED “Approved” TS_MIME_LEN_APPROVED TS_MIME_FIELD_AUTHORIZATION “Authorization” TS_MIME_LEN_AUTHORIZATION TS_MIME_FIELD_BYTES “Bytes” TS_MIME_LEN_BYTES TS_MIME_FIELD_CACHE_CONTROL “Cache-Control” TS_MIME_LEN_CACHE_CONTROL TS_MIME_FIELD_CLIENT_IP “Client-ip” TS_MIME_LEN_CLIENT_IP TS_MIME_FIELD_CONNECTION “Connection” TS_MIME_LEN_CONNECTION TS_MIME_FIELD_CONTENT_BASE “Content-Base” TS_MIME_LEN_CONTENT_BASE TS_MIME_FIELD_CONTENT_ENCODING “Content-Encoding” TS_MIME_LEN_CONTENT_ENCODING TS_MIME_FIELD_CONTENT_LANGUAGE “Content-Language” TS_MIME_LEN_CONTENT_LANGUAGE TS_MIME_FIELD_CONTENT_LENGTH “Content-Length” TS_MIME_LEN_CONTENT_LENGTH TS_MIME_FIELD_CONTENT_LOCATION “Content-Location” TS_MIME_LEN_CONTENT_LOCATION TS_MIME_FIELD_CONTENT_MD5 “Content-MD5” TS_MIME_LEN_CONTENT_MD5 TS_MIME_FIELD_CONTENT_RANGE “Content-Range” TS_MIME_LEN_CONTENT_RANGE TS_MIME_FIELD_CONTENT_TYPE “Content-Type” TS_MIME_LEN_CONTENT_TYPE TS_MIME_FIELD_CONTROL “Control” TS_MIME_LEN_CONTROL TS_MIME_FIELD_COOKIE “Cookie” TS_MIME_LEN_COOKIE TS_MIME_FIELD_DATE “Date” TS_MIME_LEN_DATE TS_MIME_FIELD_DISTRIBUTION “Distribution” TS_MIME_LEN_DISTRIBUTION TS_MIME_FIELD_ETAG “Etag” TS_MIME_LEN_ETAG
338 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TS_MIME_FIELD_EXPECT “Expect” TS_MIME_LEN_EXPECT TS_MIME_FIELD_EXPIRES “Expires” TS_MIME_LEN_EXPIRES TS_MIME_FIELD_FOLLOWUP_TO “Followup-To” TS_MIME_LEN_FOLLOWUP_TO TS_MIME_FIELD_FROM “From” TS_MIME_LEN_FROM TS_MIME_FIELD_HOST “Host” TS_MIME_LEN_HOST TS_MIME_FIELD_IF_MATCH “If-Match” TS_MIME_LEN_IF_MATCH TS_MIME_FIELD_IF_MODIFIED_SINCE “If-Modiﬁed-Since” TS_MIME_LEN_IF_MODIFIED_SINCE TS_MIME_FIELD_IF_NONE_MATCH “If-None-Match” TS_MIME_LEN_IF_NONE_MATCH TS_MIME_FIELD_IF_RANGE “If-Range” TS_MIME_LEN_IF_RANGE TS_MIME_FIELD_IF_UNMODIFIED_SINCE “If-Unmodiﬁed-Since” TS_MIME_LEN_IF_UNMODIFIED_SINCE TS_MIME_FIELD_KEEP_ALIVE “Keep-Alive” TS_MIME_LEN_KEEP_ALIVE TS_MIME_FIELD_KEYWORDS “Keywords” TS_MIME_LEN_KEYWORDS TS_MIME_FIELD_LAST_MODIFIED “Last-Modiﬁed” TS_MIME_LEN_LAST_MODIFIED TS_MIME_FIELD_LINES “Lines” TS_MIME_LEN_LINES TS_MIME_FIELD_LOCATION “Location” TS_MIME_LEN_LOCATION TS_MIME_FIELD_MAX_FORWARDS “Max-Forwards” TS_MIME_LEN_MAX_FORWARDS TS_MIME_FIELD_MESSAGE_ID “Message-ID” TS_MIME_LEN_MESSAGE_ID TS_MIME_FIELD_NEWSGROUPS “Newsgroups” TS_MIME_LEN_NEWSGROUPS TS_MIME_FIELD_ORGANIZATION “Organization” TS_MIME_LEN_ORGANIZATION TS_MIME_FIELD_PATH “Path” TS_MIME_LEN_PATH TS_MIME_FIELD_PRAGMA “Pragma” TS_MIME_LEN_PRAGMA TS_MIME_FIELD_PROXY_AUTHENTICATE “Proxy-Authenticate” TS_MIME_LEN_PROXY_AUTHENTICATE TS_MIME_FIELD_PROXY_AUTHORIZATION “Proxy-Authorization” TS_MIME_LEN_PROXY_AUTHORIZATION TS_MIME_FIELD_PROXY_CONNECTION “Proxy-Connection” TS_MIME_LEN_PROXY_CONNECTION TS_MIME_FIELD_PUBLIC “Public” TS_MIME_LEN_PUBLIC TS_MIME_FIELD_RANGE “Range” TS_MIME_LEN_RANGE TS_MIME_FIELD_REFERENCES “References” TS_MIME_LEN_REFERENCES TS_MIME_FIELD_REFERER “Referer” TS_MIME_LEN_REFERER TS_MIME_FIELD_REPLY_TO “Reply-To” TS_MIME_LEN_REPLY_TO TS_MIME_FIELD_RETRY_AFTER “Retry-After” TS_MIME_LEN_RETRY_AFTER TS_MIME_FIELD_SENDER “Sender” TS_MIME_LEN_SENDER TS_MIME_FIELD_SERVER “Server” TS_MIME_LEN_SERVER TS_MIME_FIELD_SET_COOKIE “Set-Cookie” TS_MIME_LEN_SET_COOKIE TS_MIME_FIELD_SUBJECT “Subject” TS_MIME_LEN_SUBJECTTS_MIME_LEN_SUBJECT TS_MIME_FIELD_SUMMARY “Summary” TS_MIME_LEN_SUMMARY TS_MIME_FIELD_TE “TE” TS_MIME_LEN_TE
4.7. Plugin Development 339 Apache-Trafﬁcserver-Server Documentation, latest
TS_MIME_FIELD_TRANSFER_ENCODING “Transfer-Encoding” TS_MIME_LEN_TRANSFER_ENCODING TS_MIME_FIELD_UPGRADE “Upgrade” TS_MIME_LEN_UPGRADE TS_MIME_FIELD_USER_AGENT “User-Agent” TS_MIME_LEN_USER_AGENT TS_MIME_FIELD_VARY “Vary” TS_MIME_LEN_VARY TS_MIME_FIELD_VIA “Via” TS_MIME_LEN_VIA TS_MIME_FIELD_WARNING “Warning” TS_MIME_LEN_WARNING TS_MIME_FIELD_WWW_AUTHENTICATE “Www-Authenticate” TS_MIME_LEN_WWW_AUTHENTICATE TS_MIME_FIELD_XREF “Xref” TS_MIME_LEN_XREF The header ﬁeld names above are deﬁned in ts.h as const char* strings. When Trafﬁc Server sets the name portion of a header ﬁeld (or any portion for that matter), it quickly checks to see if the new value is one of the known values. If it is, then Trafﬁc Server stores a pointer into a global table instead of storing the known value in the marshal buffer. The header ﬁeld names listed above are also pointers into this table, which enables simple pointer comparison of the value returned from TSMimeHdrFieldNameGet with one of the values listed above. It is recommended that you use the above values when referring to one of the known header ﬁeld names to avoid the possibility of a spelling error. Trafﬁc Server adds one important feature to MIME ﬁelds that you may not know about: Trafﬁc Server does not print a MIME ﬁeld if the ﬁeld name begins with the ‘@‘ symbol. For example: a plugin can add the ﬁeld “@My-Field” to a header. Even though Trafﬁc Server never sends that ﬁeld out in a request to an origin server or in a response to a client, it can be printed to Trafﬁc Server logs by deﬁning a custom log conﬁguration ﬁle that explicitly logs such ﬁelds. This provides a useful mechanism for plugins to store information about an object in one of the MIME headers associated with the object. The MIME header functions are listed below: • TSMimeHdrFieldAppend() • TSMimeHdrFieldClone() • TSMimeHdrFieldCopy() • TSMimeHdrFieldCopyValues() • TSMimeHdrFieldCreate() • TSMimeHdrFieldDestroy() • TSMimeHdrFieldLengthGet() • TSMimeHdrFieldNameGet() • TSMimeHdrFieldNameSet() • TSMimeHdrFieldNext() • TSMimeHdrFieldNextDup() • TSMimeHdrFieldValueAppend() • TSMimeHdrFieldValueAppend() • TSMimeHdrFieldValueDateGet() • TSMimeHdrFieldValueDateInsert() • TSMimeHdrFieldValueDateSet() • TSMimeHdrFieldValueIntGet() • TSMimeHdrFieldValueIntSet() • TSMimeHdrFieldValueStringGet()
340 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
• TSMimeHdrFieldValueStringInsert() • TSMimeHdrFieldValueStringSet() • TSMimeHdrFieldValueUintGet() • TSMimeHdrFieldValueUintInsert() • TSMimeHdrFieldValueUintSet() • TSMimeHdrFieldValuesClear() • TSMimeHdrFieldValuesCount() • TSMimeHdrClone() • TSMimeHdrCopy() • TSMimeHdrCreate() • TSMimeHdrDestroy() • TSMimeHdrFieldFind() • TSMimeHdrFieldGet() • TSMimeHdrFieldRemove() • TSMimeHdrFieldsClear() • TSMimeHdrFieldsCount() • TSMimeHdrLengthGet() • TSMimeHdrParse() • TSMimeParserClear() • TSMimeParserCreate() • TSMimeParserDestroy() • TSMimeHdrPrint()
Marshal Buffers
A marshal buffer, or TSMBuffer, is a heap data structure that stores parsed URLs, MIME headers, and HTTP headers. You can allocate new objects out of marshal buffers and change the values within a marshal buffer. Whenever you manipulate an object, you require the handle to the object (TSMLoc) and the marshal buffer containing the object (TSMBuffer). Routines exist for manipulating the object based on these two pieces of information. For examples, see one of the following: • HTTP Headers • URLs • MIME Headers The marshal buffer functions enable you to create and destroy Trafﬁc Server’s marshal buffers, which are the data structures that hold parsed URLs, MIME headers, and HTTP headers.
: Any marshal buffer fetched by TSHttpTxn*Get will be used by other parts of the system. Be careful not to destroy these shared transaction marshal buffers in functions such as those below:
4.7. Plugin Development 341 Apache-Trafﬁcserver-Server Documentation, latest
• TSHttpTxnCachedReqGet • TSHttpTxnCachedRespGet • TSHttpTxnClientReqGet • TSHttpTxnClientRespGet • TSHttpTxnServerReqGet • TSHttpTxnServerRespGet • TSHttpTxnTransformRespGet
URLs
API URL functions provide access to URL data stored in marshal buffers. The URL functions can create, copy, retrieve or delete entire URLs; they can also retrieve or modify parts of URLs, such as port or scheme information. The general form of an Internet URL is:
 scheme://user:password@host:port/stuff
The URL data structure includes support for two speciﬁc types of internet URLs. HTTP URLs have the form:
 http://user:password@host:port/path;params?query#fragment
The URL port is stored as integer. All remaining parts of the URL (scheme, user, etc.) are stored as strings. Trafﬁc Server URL functions are named according to the portion of the URL on which they operate. For instance, the function that retrieves the host portion of a URL is named TSUrlHostGet. To facilitate fast comparisons and reduce storage size, Trafﬁc Server deﬁnes several preallocated scheme names. TS_URL_SCHEME_FILE “ﬁle” TS_URL_LEN_FILE TS_URL_SCHEME_FTP “ftp” TS_URL_LEN_FTP TS_URL_SCHEME_GOPHER “gopher” TS_URL_LEN_GOPHER TS_URL_SCHEME_HTTP “http” TS_URL_LEN_HTTP TS_URL_SCHEME_HTTPS “https” TS_URL_LEN_HTTPS TS_URL_SCHEME_MAILTO “mailto” TS_URL_LEN_MAILTO TS_URL_SCHEME_NEWS “news” TS_URL_LEN_NEWS TS_URL_SCHEME_NNTP “nntp” TS_URL_LEN_NNTP TS_URL_SCHEME_PROSPERO “prospero” TS_URL_LEN_PROSPERO TS_URL_SCHEME_TELNET “telnet” TS_URL_LEN_TELNET TS_URL_SCHEME_WAIS “wais” TS_URL_LEN_WAIS The scheme names above are deﬁned in ts.h as const char* strings. When Trafﬁc Server sets the scheme portion of the URL (or any portion for that matter), it quickly checks to see if the new value is one of the known values. If it is, then it stores a pointer into a global table (instead of storing the known value in the marshal buffer). The scheme values listed above are also pointers into this table. This allows simple pointer comparison of the value returned from TSUrlSchemeGet with one of the values listed above. You should use the Trafﬁc Server-deﬁned values when referring to one of the known schemes, since doing so can prevent the possibility of spelling errors. Trafﬁc Server URL functions are listed below:
342 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSUrlClone() TSUrlCopy() TSUrlCreate() TSUrlDestroy() TSUrlPrint() TSUrlFtpTypeGet() TSUrlFtpTypeSet() TSUrlHostGet() TSUrlHostSet() TSUrlHttpFragmentGet() TSUrlHttpFragmentSet() TSUrlHttpParamsGet() TSUrlHttpParamsSet() TSUrlHttpQueryGet() TSUrlHttpQuerySet() TSUrlLengthGet() TSUrlParse() TSUrlPasswordGet() TSUrlPasswordSet() TSUrlPathGet() TSUrlPathSet() TSUrlPortGet() TSUrlPortSet() TSUrlSchemeGet() TSUrlSchemeSet() TSUrlStringGet() TSUrlUserGet() TSUrlUserSet() An HTTP message consists of the following: • HTTP header • body • trailer The HTTP header consists of: • A request or response line – An HTTP request line contains a method, URL, and version – A response line contains a version, status code, and reason phrase • A MIME header A MIME header is comprised of zero or more MIME ﬁelds. A MIME ﬁeld is composed of a ﬁeld name, a colon, and (zero or more) ﬁeld values. The values in a ﬁeld are separated by commas. An HTTP header containing a request line is usually referred to as a request. The following example shows a typical request header:
GET http://www.tiggerwigger.com/ HTTP/1.0 Proxy-Connection: Keep-Alive User-Agent: Mozilla/5.0 [en] (X11; I; Linux 2.2.3 i686) Host: www.tiggerwigger.com Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */* Accept-Encoding: gzip Accept-Language: en Accept-Charset: iso-8859-1, *, utf-8
The response header for the above request might look like the following:
HTTP/1.0 200OK Date: Fri, 13 Nov 2009 06:57:43 GMT Content-Location: http://locutus.tiggerwigger.com/index.html Etag: "07db14afa76be1:1074" Last-Modified: Thu, 05 Nov 2009 20:01:38 GMT Content-Length: 7931 Content-Type: text/html Server: Microsoft-IIS/4.0 Age: 922 Proxy-Connection: close
The following ﬁgure illustrates an HTTP message with an expanded HTTP header. Figure 10.1. HTTP Request/Response and Header Structure The ﬁgure below shows example HTTP request and response headers. Figure 10.2. Examples of HTTP Request and Response Headers The marshal buffer or TSMBuffer is a heap data structure that stores parsed URLs, MIME headers, and HTTP head- ers. You can allocate new objects out of marshal buffers and change the values within the marshal buffer. Whenever
4.7. Plugin Development 343 Apache-Trafﬁcserver-Server Documentation, latest
4.14: HTTP Request/Response and Header Structure
4.15: Examples of HTTP Request and Response Headers
344 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 you manipulate an object, you must require the handle to the object (TSMLoc) and the marshal buffer containing the object (TSMBuffer). Figure 10.3. Marshal Buffers and Header Locations
4.16: Marshal Buffers and Header Locations
The ﬁgure above shows the following: • The marshal buffer containing the HTTP request, reqest_bufp • TSMLoc location pointer for the HTTP header (http_hdr_loc) • TSMLoc location pointer for the request URL (url_loc) • TSMLoc location pointers for the MIME header (mime_hdr_loc) • TSMLoc location pointers for MIME ﬁelds (fieldi_loc) • TSMLoc location pointer for the next duplicate MIME ﬁeld (next_dup_loc) The diagram also shows that an HTTP header contains pointers to the URL location and the MIME header location. You can obtain the URL location from an HTTP header using the function TSHttpHdrUrlGet. To work with MIME headers, you can pass either a MIME header location or an HTTP header location to MIME header functions
4.7. Plugin Development 345 Apache-Trafﬁcserver-Server Documentation, latest
. If you pass an HTTP header to a MIME header function, then the system locates the associated MIME header and executes the MIME header function on the MIME header location.
HTTP Transformations
Transform plugins examine or transform HTTP message body content. For example, transform plugins can: • Append text to HTML documents • Compress images • Do virus checking (on client POST data or server response data) • Do content-based ﬁltering (ﬁlter out HTML documents that contain certain terms or expressions) This chapter explains how to write transform plugins. The following examples are discussed in detail:
Sample Null Transform Plugin
This section provides a step-by-step description of what the null transform plugin does, along with sections of code that apply. For context, you can ﬁnd each code snippet in the complete source code. Some of the error checking details are left out - to give the description a step-by-step ﬂow, only the highlights of the transform are included. Below is an overview of the null transform plugin: 1. Gets a handle to HTTP transactions.
 void TSPluginInit( int argc, const char *argv[]) { TSHttpHookAdd (TS_HTTP_READ_RESPONSE_HDR_HOOK, TSContCreate (transform_plugin, NULL));
With this TSPluginInit routine, the plugin is called back every time Trafﬁc Server reads a response header. 2. Checks to see if the transaction response is transformable.
 static int transform_plugin (TSCont contp, TSEvent event, void *edata) { TSHttpTxn txnp= (TSHttpTxn) edata; switch (event) { case TS_EVENT_HTTP_READ_RESPONSE_HDR: if (transformable (txnp)) { transform_add (txnp); }
The default behavior for transformations is to cache the transformed content (you can also tell Trafﬁc Server to cache untransformed content, if you want). Therefore, only responses received directly from an origin server need to be transformed. Objects served from cache are already transformed. To determine whether the response is from the origin server, the routine transformable checks the response header for the “200 OK” server response.
 static int transformable (TSHttpTxn txnp) { TSMBuffer bufp; TSMLoc hdr_loc; TSHttpStatus resp_status; TSHttpTxnServerRespGet (txnp,&bufp,&hdr_loc);
 if (TS_HTTP_STATUS_OK== (resp_status=
346 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSHttpHdrStatusGet (bufp, hdr_loc)) ) { return 1; } else { return 0; } }
3. If the response is transformable, then the plugin creates a transformation vconnection that gets called back when the response data is ready to be transformed (as it is streaming from the origin server).
 static void transform_add (TSHttpTxn txnp) { TSVConn connp; connp= TSTransformCreate (null_transform, txnp); TSHttpTxnHookAdd (txnp, TS_HTTP_RESPONSE_TRANSFORM_HOOK, connp); }
The previous code fragment shows that the handler function for the transformation vconnection is null_transform. 4. Get a handle to the output vconnection (that receives data from the tranformation).
 output_conn= TSTransformOutputVConnGet (contp);
5. Get a handle to the input VIO. (See the handle_transform function.)
 input_vio= TSVConnWriteVIOGet (contp);
This is so that the transformation can get information about the upstream vconnection’s write operation to the input buffer. 6. Initiate a write to the output vconnection of the speciﬁed number of bytes. When the write is initiated, the trans- formation expects to receive WRITE_READY, WRITE_COMPLETE, or ERROR events from the output vconnec- tion. See the handle_transform function for the following code fragment:
 data->output_vio= TSVConnWrite (output_conn, contp, data->output_reader, TSVIONBytesGet (input_vio));
7. Copy data from the input buffer to the output buffer. See the handle_transform function for the following code fragment:
TSIOBufferCopy (TSVIOBufferGet (data->output_vio), TSVIOReaderGet (input_vio), towrite,0);
8. Tell the input buffer that the transformation has read the data. See the handle_transform function for the following code fragment:
TSIOBufferReaderConsume (TSVIOReaderGet (input_vio), towrite);
9. Modify the input VIO to tell it how much data has been read (increase the value of ndone). See the handle_transform function for the following code fragment:
TSVIONDoneSet (input_vio, TSVIONDoneGet (input_vio)+ towrite);
10. If there is more data left to read ( if ndone < nbytes), then the handle_transform function wakes up the downstream vconnection with a reenable and wakes up the upstream vconnection by sending it WRITE_READY:
4.7. Plugin Development 347 Apache-Trafﬁcserver-Server Documentation, latest
 if (TSVIONTodoGet (input_vio)>0){ if (towrite>0){ TSVIOReenable (data->output_vio);
TSContCall (TSVIOContGet (input_vio), TS_EVENT_VCONN_WRITE_READY, input_vio); } } else {
The process of passing data through the transformation is illustrated in the following diagram. The downstream vconnections send WRITE_READY events when they need more data; when data is available, the upstream vconnections reenable the downstream vconnections. In this instance, the TSVIOReenable function sends TS_EVENT_IMMEDIATE. Passing Data Through a Transformation {#PassingDataThroughaTransformation}
4.17: Passing Data Through a Transformation
11. If the handle_transform function ﬁnds there is no more data to read, then it sets nbytes to ndone on the output (downstream) VIO and wakes up the output vconnection with a reenable. It then triggers the end of the write operation from the upstream vconnection by sending the upstream vconnection a WRITE_COMPLETE event.
TSVIONBytesSet (data->output_vio, TSVIONDoneGet (input_vio)); TSVIOReenable (data->output_vio); TSContCall (TSVIOContGet (input_vio), TS_EVENT_VCONN_WRITE_COMPLETE, input_vio); }
When the upstream vconnection receives the WRITE_COMPLETE event, it will probably shut down the write operation. 12. Similarly, when the downstream vconnection has consumed all of the data, it sends the transformation a WRITE_COMPLETE event. The transformation handles this event with a shut down (the transformation shuts down the write operation to the downstream vconnection). See the null_plugin function for the following code fragment:
 case TS_EVENT_VCONN_WRITE_COMPLETE: TSVConnShutdown (TSTransformOutputVConnGet (contp),0,1 break;
348 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
The following diagram illustrates the ﬂow of events: Ending the Transformation {#EndingTransformation}
4.18: Ending the Transformation
Append-Transform Plugin
The append-transform plugin appends text to the body of an HTTP response. It obtains this text from a ﬁle; the name of the ﬁle containing the append text is a parameter you specify in plugin.config, as follows: append-transform.so path/to/file
The append-transform plugin is based on null-transform.c. The only difference is that after the plugin feeds the document through the transformation, it adds text to the response. Below is a list of the functions in append-transform.c, in the order they appear in the source code. Below each entry is a description of what the function does: • ‘‘my_data_alloc‘‘ Allocates and initializes a MyData structure. The plugin deﬁnes a struct, MyData, as follows:
 typedef struct { TSVIO output_vio; TSIOBuffer output_buffer; TSIOBufferReader output_reader; int append_needed; } MyData;
The MyData structure is used to represent data that the transformation (vconnection) needs. The transforma- tion’s data pointer is set to a MyData pointer using TSContDataSet in the handle_transform routine. • ‘‘my_data_destroy‘‘ Destroys objects of type MyData. To deallocate the transform’s data, the append_transform routine (see below) calls my_data_destroy when the transformation is complete. • ‘‘handle_transform‘‘
4.7. Plugin Development 349 Apache-Trafﬁcserver-Server Documentation, latest
This function does the actual data transformation. The transformation is created in transform_add (see below). handle_transform is called by append_transform. • ‘‘append_transform‘‘ This is the handler function for the transformation vconnection created in transform_add. It is the imple- mentation of the vconnection. – If the transformation vconnection has been closed, then append_transform calls my_data_destroy to destroy the vonnection. – If append_transform receives an error event, then it calls back the continuation to let it know it has completed the write operation. – If it receives a WRITE_COMPLETE event, then it shuts down the write portion of its vconnection. – If it receives a WRITE_READY or any other event (such as TS_HTTP_RESPONSE_TRANSFORM_HOOK), then it calls handle_transform to attempt to transform more data. • ‘‘transformable‘‘ The plugin transforms only documents that have a content type of text/html. This function examines the Content-Type MIME header ﬁeld in the response header. If the value of the MIME ﬁeld is text/html, then the function returns 1; otherwise, it returns zero. • ‘‘transform_add‘‘ Creates the transformation for the current transaction and sets up a transformation hook. The handler function for the transformation is append_transform. • ‘‘transform_plugin‘‘ This is the handler function for the main continuation for the plugin. Trafﬁc Server calls this function whenever it reads an HTTP response header. transform_plugin does the following: – Gets a handle to the HTTP transaction being processed – Calls transformable to determine whether the response document content is of type text/html – If the content is transformable, then it calls transform_add to create the transformation. – Calls TSHttpTxnReenable to continue the transaction • ‘‘load‘‘ Opens the ﬁle containing the text to be appended and loads the contents of the ﬁle into an TSIOBuffer called append_buffer. • ‘‘TSPluginInit‘‘ Does the following: – Checks to make sure that the required conﬁguration information (the append text ﬁlename) is entered in plugin.config correctly. – If there is a ﬁlename, then TSPluginInit calls load to load the text. – Creates a continuation for the plugin. The handler for this continuation is transform_plugin. – Adds the plugin’s continuation to TS_HTTP_READ_RESPONSE_HDR_HOOK. In other words, it sets up a callback of the plugin’s continuation when Trafﬁc Server reads HTTP response headers.
350 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Sample Buffered Null Transform Plugin
The buffered null transform, bnull-transform.c, reads the response content into a buffer and then writes the full buffer out to the client. Many examples of transformations, such as compression, require you to gather the full response content in order to perform the transformation. The buffered null transform uses a state variable to keep track of when it is (a) reading data into the buffer and (b) writing the data from the buffer to the downstream vconnection. The following is a step-by-step walk through the buffered null transform: 1. Gets a handle to HTTP transactions.
 void TSPluginInit( int argc, const char *argv[]) { TSHttpHookAdd (TS_HTTP_READ_RESPONSE_HDR_HOOK, TSContCreate (transform_plugin, NULL)); }
With this TSPluginInit routine, the plugin is called back every time Trafﬁc Server reads a response header. 2. Checks to see if the transaction response is transformable.
 static int transform_plugin (TSCont contp, TSEvent event, void *edata) { TSHttpTxn txnp= (TSHttpTxn) edata; switch (event) { case TS_EVENT_HTTP_READ_RESPONSE_HDR: if (transformable (txnp)) { transform_add (txnp); }
The default behavior for transformations is to cache the transformed content (if desired, you also can tell Trafﬁc Server to cache untransformed content). Therefore, only responses received directly from an origin server need to be transformed. Objects served from the cache are already transformed. To determine whether the response is from the origin server, the routine transformable checks the response header for the “200 OK” server response.
{ TSMBuffer bufp; TSMLoc hdr_loc; TSHttpStatus resp_status;
TSHttpTxnServerRespGet (txnp,&bufp,&hdr_loc);
 if(TS_HTTP_STATUS_OK== (resp_status=TSHttpHdrStatusGet(bufp,hdr_loc))) { return 1; } else { return 0; } }
3. If the response is transformable, then the plugin creates a transformation vconnection that gets called back when the response data is ready to be transformed (as it is streaming from the origin server).
 static void transform_add (TSHttpTxn txnp) { TSVConn connp; connp= TSTransformCreate (bnull_transform, txnp);
4.7. Plugin Development 351 Apache-Trafﬁcserver-Server Documentation, latest
TSHttpTxnHookAdd (txnp, TS_HTTP_RESPONSE_TRANSFORM_HOOK, connp); }
The previous code fragment shows that the handler function for the transformation vconnection is bnull_transform. 4. The bnull_transform function has to handle ERROR, WRITE_COMPLETE, WRITE_READY, and IMMEDIATE events. If the transform is just beginning, the event received is probably IMMEDIATE. The bnull_transform function calls handle_transform to handle WRITE_READY and IMMEDIATE. 5. The handle_transform function examines the data parameter for the continuation passed to it (the contin- uation passed to handle_transform is the transformation vconnection). The data structure keeps track of two states: copying the data into the buffer (STATE_BUFFER_DATA) and writing the contents of the buffer to the output vconnection (STATE_OUTPUT_DATA). 6. Get a handle to the input VIO (see the handle_buffering function). input_vio = TSVConnWriteVIOGet (contp); This is so that the transformation can get information about the up- stream vconnection’s write operation to the input buffer. 7. Copy data from the input buffer to the output buffer. See the handle_buffering function for the following code fragment:
TSIOBufferCopy (data->output_buffer, TSVIOReaderGet (write_vio), towrite,0);
8. Tell the input buffer that the transformation has read the data. See the handle_buffering function for the following code fragment:
TSIOBufferReaderConsume (TSVIOReaderGet (write_vio), towrite);
9. Modify the input VIO to tell it how much data has been read (increase the value of ndone). See the handle_buffering function for the following code fragment:
TSVIONDoneSet (write_vio, TSVIONDoneGet (write_vio)+ towrite); }
10. If there is more data left to read ( if ndone < nbytes), then the handle_buffering function wakes up the upstream vconnection by sending it WRITE_READY:
 if (TSVIONTodoGet (write_vio)>0){ if (towrite>0){ TSContCall (TSVIOContGet (write_vio), TS_EVENT_VCONN_WRITE_READY, write_vio); } } else {
The process of passing data through the transformation is illustrated in the following diagram. The transforma- tion sends WRITE_READY events when it needs more data; when data is available, the upstream vconnection reenables the transformation with an IMMEDIATE event. The following diagram illustrates the read from an input vconnection: Reading Data Into the Buffer (the ‘‘STATE_BUFFER_DATA‘‘ State) {#ReadingDataIntoBuffer} 11. When the data is read into the output buffer, the handle_buffering function sets the state of the trans- formation’s data structure to STATE_OUTPUT_DATA and calls the upstream vconnection back with the WRITE_COMPLETE event.
352 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
4.19: Reading Data Into the Buffer the STATE_BUFFER_DATA State
 data->state= STATE_OUTPUT_DATA; TSContCall (TSVIOContGet (write_vio), TS_EVENT_VCONN_WRITE_COMPLETE, write_vio);
12. The upstream vconnection will probably shut down the write operation when it receives the WRITE_COMPLETE event. The handler function of the transformation, bnull_transform, receives an IMMEDIATE event and calls the handle_transform function. This time, the state is STATE_OUTPUT_DATA, so handle_transform calls handle_output. 13. The handle_output function gets a handle to the output vconnection: output_conn = TSTransformOutputVConnGet (contp); 14. The handle_output function writes the buffer to the output vconnection:
 data->output_vio= TSVConnWrite (output_conn, contp, data->output_reader, TSIOBufferReaderAvail (data->output_reader) );
The following diagram illustrates the write to the output vconnection: Writing the Buffered Data to the Output Vconnection {#WritingBufferedtDataIntoVConnection}
Writing Content Transform Plugins
Content transformation plugins transform HTTP response content (such as images or HTML documents) and HTTP request content (such as client POST data). Because the data stream to be transformed is of variable length, these plugins must use a mechanism that passes data from buffer to buffer and checks to see if the end of the data stream is reached. This mechanism is provided by virtual connections (VConnections) and virtual IO descriptors (VIOs). A VConnection is an abstraction for a data pipe that allows its users to perform asynchronous reads and writes without knowing the underlying implementation. A transformation is a speciﬁc type of VConnection.A transfor- mation connects an input data source and an output data sink; this feature enables it to view and modify all the data passing through it.
4.7. Plugin Development 353 Apache-Trafﬁcserver-Server Documentation, latest
4.20: Writing the Buffered Data to the Output Vconnection
Transformations can be chained together, one after the other, so that multiple transformations can be performed on the same content. The VConnection type, TSVConn, is actually a subclass of TSCont, which means that VConnections (and transformations) are continuations. VConnections and transformations can thus exchange events, informing one another that data is available for reading or writing, or that the end of a data stream is reached. A VIO is a description of an IO operation that is in progress. Every VConnection has an associated input VIO and an associated output VIO. When VConnections are transferring data to one another, one VConnection‘s input VIO is another VConnection‘s output VIO.A VConnection‘s input VIO is also called its write ‘‘VIO‘‘ because the input VIO refers to a write operation performed on the VConnection itself. Similarly, the output VIO is also called the read ‘‘VIO‘‘. For transformations, which are designed to pass data in one direction, you can picture the relationship between the transformation VConnection and its VIOs as follows:
4.21: A Transformation and its VIOs
Because the Trafﬁc Server API places transformations directly in the response or request data stream, the transforma- tion VConnection is responsible only for reading the data from the input buffer, transforming it, and then writing it to the output buffer. The upstream VConnection writes the incoming data to the transformation’s input buffer. In the ﬁgure above, A Transformation and its VIOs, the input VIO describes the progress of the upstream VConnection‘s write operation on the transformation, while the output VIO describes the progress of the transformation’s write op- eration on the output (downstream) VConnection. The nbytes value in the VIO is the total number of bytes to be written. The ndone value is the current progress, or the number of bytes that have been written at a speciﬁc point in
354 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest time. When writing a transformation plugin, you must understand implementation as well as the use of VConnections. The implementor’s side refers to how to implement a VConnection that others can use. At minimum, a transform plugin creates a transformation that sits in the data stream and must be able to handle the events that the upstream and downstream VConnections send to it. The user’s side refers to how to use a VConnection to read or write data. At the very least, transformations output (write) data.
Transformations
VIOs
A VIO``*or virtual IO is a description of an in progress IO operation. The ``VIO data structure is used by VConnection users to determine how much progress has been made on a particu- lar IO operation, and to reenable an IO operation when it stalls due to buffer space. VConnection implementors use VIOs to determine the buffer for an IO operation, how much work to do on the IO operation, and which continuation to call back when progress on the IO operation is made. The TSVIO data structure itself is opaque, but it might have been deﬁned as follows: typedef struct { TSCont continuation; TSVConn vconnection; TSIOBufferReader reader; TSMutex mutex; int nbytes; int ndone; } *TSVIO;
IO Buffers
The IO buffer data structure is the building block of the VConnection abstraction. An IO buffer is composed of a list of buffer blocks which, in turn, point to buffer data. Both the buffer block (TSIOBufferBlock) and buffer data (TSIOBufferData) data structures are reference counted so they can reside in multiple buffers at the same time. This makes it extremely efﬁcient to copy data from one IO buffer to another using TSIOBufferCopy, since Trafﬁc Server only needs to copy pointers and adjust reference counts appropriately (instead of actually copying any data). The IO buffer abstraction provides for a single writer and multiple readers. In order for the readers to have no knowl- edge of each other, they manipulate IO buffers through theTSIOBufferReader data structure. Since only a single writer is allowed, there is no corresponding TSIOBufferWriter data structure. The writer simply modiﬁes the IO buffer directly.
Transaction Data Sink
The hook TS_HTTP_RESPONSE_CLIENT_HOOK is a hook that supports a special type of transformation, one with only input and no output. Although the transformation doesn’t provide data back to Trafﬁc Server it can do anything else with the data, such as writing it to another output device or process. It must, however, consume all the data for the transaction. There are two primary use cases. 1. Tap in to the transaction to provide the data for external processing. 2. Maintain the transaction.
4.7. Plugin Development 355 Apache-Trafﬁcserver-Server Documentation, latest
For the latter it is important to note that if all consumers of a transaction (primarily the user agent) shut down the transaction is also terminated, including the connection to the origin server. A data sink transform, unlike a standard transform, is considered to be a consumer and will keep the transaction and the origin server connection up. This is useful when the transaction is in some way expensive and should run to completion even if the user agent disconnects. Examples would be a standard transform that is expensive to initiate, or expensive origin server connections that should be :ts:cv:‘shared ‘. There is an example plugin that demonstrates this used as a pure data sink to keep the transaction up regardless of whether the user agent disconnects.
New Protocol Plugins
The new protocol APIs enable you to extend Trafﬁc Server to be a web proxy for any protocol. This chapter describes new protocol APIs and the plugins that support new protocols. It also provides a detailed review of code for a sample Protocol plugin that supports a very simple artiﬁcial HTTP-like protocol.
About the Sample Protocol
The sample protocol enables a client to ask a server for a ﬁle. Clients send requests to a speciﬁc Trafﬁc Server port (speciﬁed in plugin.config); each request has the following structure:
 server_name file_name
Using the Protocol plugin, Trafﬁc Server can accept these requests, parse them, and act as a proxy cache (i.e., request the ﬁle from the origin server on the client’s behalf and store copies of response messages in cache). The Protocol plugin is a state machine that ﬂows through the states illustrated in the Sample Protocol State Diagram. This ﬁgure illustrates the steps that Trafﬁc Server and the Protocol plugin go through in order to support the sample protocol. In more speciﬁc terms, Trafﬁc Server and the Protocol plugin must: • Listen for and accept client connections (on the accept port speciﬁed in plugin.config) • Read incoming client requests • Look up the requested content in the Trafﬁc Server cache • Serve content from cache if the request is a cache hit (this simple example does not do freshness checking) • Open a connection to the origin server if the request is a cache miss (on the server port speciﬁed in plugin. config) • Forward the request to the origin server • Receive the origin server response • Cache the response and send it on to the client Sample Protocol State Diagram
Protocol Plugin Structure
To see how the Protocol plugin works, you need to understand some broader concepts. This section assumes you’re familiar with the concepts of continuation, Trafﬁc Server’s asynchronous event model, and basic Trafﬁc Server plugin structure. If you are not familiar with these concepts, then you may want to begin with the Getting Started section.
356 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
4.22: Sample Protocol State Diagram
4.7. Plugin Development 357 Apache-Trafﬁcserver-Server Documentation, latest
Continuations in the Protocol Plugin
The Protocol plugin creates a static continuation that is an “accept” state machine - that is, a state machine whose job is to accept client connections on the appropriate port. When Trafﬁc Server accepts a net connection from a client on that port, the accept state machine is activated. It then creates a new continuation: a transaction state machine. The accept state machine creates one transaction state machine for each transaction (where a transaction consists of a client request and Trafﬁc Server’s response). Each transaction state machine lives until the transaction completes; then it is destroyed. If the client’s request for content is a cache miss, then a transaction state machine might need to open a connection to the origin server. This is illustrated in the Protocol Plugin Overview diagram below. Protocol Plugin Overview
4.23: Protocol Plugin Overview
The ﬁrst steps for writing the Protocol plugin are now clear: in TSPluginInit, you must create a continuation that listens for net connections on the client port speciﬁed in plugin.config (this continuation is the accept state machine). Below is a summary of the continuations implemented for the Protocol plugin: • An accept state machine that listens for client connections, and then creates transaction state machines when- ever Trafﬁc Server accepts a new client connection. The accept state machine lives as long as Trafﬁc Server is running. • Transaction state machines that read client requests, process them, and are then destroyed when the transaction is ﬁnished.
Event Flow
Implementing the rest of the Protocol plugin requires that you understand the ﬂow of events during the course of a transaction. Unlike HTTP transaction plugins, this plugin must read data from network connections and then read/write data to the Trafﬁc Server cache. This means that its continuations do not receive HTTP state machine events; they receive events from Trafﬁc Server’s processor subsystems. For example: the accept state machine is activated by an TS_EVENT_NET_ACCEPT event from Trafﬁc Server’s Net Processor; the handler function for the accept state machine must therefore be able to handle that event.
358 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
The transaction state machines are activated when the client connection receives incoming request data. The Net Processor notiﬁes the transaction state machine of incoming data. The transaction state machine reads the data; when ﬁnished, it initiates a cache lookup of the requested ﬁle. When the cache lookup completes, the transaction state machine is activated by the Trafﬁc Server Cache Processor. If the transaction state machine needs to open a connection to the origin server to fetch content (in the case of a cache miss), then the transaction state machine initiates a DNS lookup of the server name. The transaction state machine is activated by a DNS lookup event from the Trafﬁc Server Host Database Processor. If the transaction must connect to the origin server, then the transaction state machine initiates a net connection and waits for an event from the Net Processor. Protocol Plugin Flow of Events
4.24: Protocol Plugin Flow of Events
The ﬂow of events is illustrated in the Protocol Plugin Flow of Events diagram above. The thin straight lines show Net Processor event ﬂow, the thin dashed lines represent Host Database event ﬂow, and the thick dashed lines show Cache event ﬂow. Notice that this ﬂow of events is independent of the Protocol plugin’s design (i.e., whether you build accept or transaction state machines). Any plugin that supports network connections uses the net vconnection interfaces (TSNetAccept, TSNetConnect) and thus receives events from the Net Processor. Any plugin that performs cache lookups or cache writes uses TSCacheRead, TSCacheWrite, TSVConnRead, and TSVConnWrite and thus receives events from the Cache Processor and Trafﬁc Server event system. Similarly, any plugin that does DNS lookups receives events from the Host Database Processor.
One Way to Implement a Transaction State Machine
Transaction state machines (TSMs) in the Protocol plugin must do the following: • Keep track of the state of the transaction • Handle events received (based on the state of the transaction and the event received) • Update the state of the transaction as it changes Below is one way you can implement TSMs. Details about how the Protocol plugin does this are provided in the next section. • Create a data structure for transactions that contains all of the state data you need to keep track of. In the Protocol plugin this is a struct, Txn_SM.
4.7. Plugin Development 359 Apache-Trafﬁcserver-Server Documentation, latest
• When you create the TSM’s continuation, initialize data of type Txn_SM. Initialize the data to the initial state of a transaction (in this case, a net connection has just been accepted). Associate this data to the TSM continuation using TSContDataSet. • Write state handler functions that handle the expected events for each state. • Write the handler for the TSM. Its job is to receive events, examine the current state, and execute the appropriate state handler function. In the Protocol plugin, the handler is main_handler. main_handler calls the state handler functions to handle each state. The steps below describe the ﬂow of execution illustrated in “How Transaction State Machines are Implemented in the Protocol Plugin”. 1. The handler for the TSM, (called main_handler in the Protocol plugin) receives events from the TSM. 2. main_handler examines the state of the transaction-in particular, it examines the current handler. 3. main_handler calls the current_handler (which is one of the state handler functions), and then passes the current event to current_handler. In the image below below, the current handler is called state2_handler. 4. The current_handler handles the event and updates the data. In the image below below, the state is changed from state2 to state3 (and the current handler is changed from state2_handler to state3_handler). The next time main_handler receives an event, it will be processed by state3_handler. 5. state2_handler arranges the next callback of the TSM. Typically, it gives Trafﬁc Server additional work to do (such as writing a ﬁle to cache) so that it can progress to the next state. The TSM (main_handler) then waits for the next event to arrive from Trafﬁc Server. How Transaction State Machines are Implemented in the Protocol Plugin
Processing a Typical Transaction
The code is contained in the following ﬁles: • Protocol.c and Protocol.h • Accept.c and Accept.h • TxnSM.c and TxnSM.h Below is a step-by-step walk-through of the code that processes a typical transaction. 1. The TSPluginInit function is in the Protocol.c ﬁle. It checks the validity of the plugin.config entries (there must be two: a client accept port and a server port) and runs an initialization routine, init. 2. The init function (in Protocol.c) creates the plugin’s log ﬁle using TSTextLogObjectCreate. 3. The init function creates the accept state machine using AcceptCreate. The code for AcceptCreate is in the Accept.c ﬁle. 4. The accept state machine, like the transaction state machine, keeps track of its state with a data structure. This data structure, Accept, is deﬁned in the Accept.h ﬁle. State data in AcceptCreate is associated with the new accept state machine via TSContDataSet. 5. The init function arranges the callback of the accept state machine when there is a network connection by using TSNetAccept. 6. The handler for the accept state machine is accept_event in the Accept.c ﬁle. When Trafﬁc Server’s Net Processor sends TS_EVENT_NET_ACCEPT to the accept state machine, accept_event creates a transaction state machine (txn_sm) by calling TxnSMCreate. Notice that accept_event creates a mutex for the transaction state machine, since each transaction state machine has its own mutex.
360 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
4.25: How Transaction State Machines are Implemented in the Protocol Plugin
4.7. Plugin Development 361 Apache-Trafﬁcserver-Server Documentation, latest
7. The TxnSMCreate function is in the TxnSM.c ﬁle. The ﬁrst thing it does is initialize the transaction’s data, which is of type TxnSM (as deﬁned in TxnSM.h). Notice that the current handler (q_current_handler) is set to state_start. 8. TxnSMCreate then creates a transaction state machine using TSContCreate. The handler for the transac- tion state machine is main_handler, which is in the TxnSM.c ﬁle. 9. When accept_event receives TS_EVENT_NET_ACCEPT, it calls the transaction state machine ( TSContCall (txn_sm, 0, NULL); ). The event passed to main_handler is 0 (TS_EVENT_NONE). 10. The ﬁrst thing main_handler does is examine the current txn_sm state by calling TSContDataGet. The state is state_start. 11. main_handler then invokes the handler for state_start by using the function pointer TxnSMHandler (as deﬁned in TxnSM.h). 12. The state_start handler function (in the TxnSM.c ﬁle) is handed an event (at this stage, the event is TS_EVENT_NET_ACCEPT) and a client vconnection. state_start checks to see if this client vconnec- tion is closed; if it is not, then state_start attempts to read data from the client vconnection into an TSIOBuffer (state_start is handling the event it receives). 13. state_start changes the current handler to state_interface_with_client (that is, it updates the state of the transaction to the next state). 14. state_start initiates a read of the client vconnection (arranges for Trafﬁc Server to send TS_EVENT_VCONN_READ_READY events to the TSM) by calling TSVConnRead. 15. state_interface_with_client is activated by the next event from Trafﬁc Server. It checks for errors and examines the read VIO for the read operation initiated by TSVConnRead. 16. If the read VIO is the client_read_VIO (which we are expecting at this stage in the transaction), then state_interface_with_client updates the state to state_read_request_from_client . 17. state_read_request_from_client handles actual TS_EVENT_READ_READY events and reads the client request. 18. state_read_request_from_client parses the client request. 19. state_read_request_from_client updates the current state to the next state, state_handle_cache_lookup . 20. state_read_request_from_client arranges for Trafﬁc Server to call back the TSM with the next set of events (initiating the cache lookup) by calling TSCacheRead. 21. When the TSCacheRead sends the TSM either TS_EVENT_OPEN_READ (a cache hit) or TS_EVENT_OPEN_READ_FAILED (a cache miss), main_handler calls state_handle_cache_lookup.
Plugin Interfaces
Most of the functions in the Trafﬁc Server API provide an interface to speciﬁc code modules within Trafﬁc Server. The miscellaneous functions described in this chapter provide some useful general capabilities. They are categorized as follows: The C library already provides functions such as printf, malloc, and fopen to perform these tasks. The Trafﬁc Server API versions, however, overcome various C library limitations (such as portability to all Trafﬁc Server-support platforms).
362 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSfopen Family
The fopen family of functions in C is normally used for reading conﬁguration ﬁles, since fgets is an easy way to parse ﬁles on a line-by-line basis. The TSfopen family of functions aims at solving the same problem of buffered IO and line at a time IO in a platform-independent manner. The fopen family of C library functions can only open a ﬁle if a ﬁle descriptor less than 256 is available. Since Trafﬁc Server often has more than 2000 ﬁle descriptors open at once, however, the likelihood of an available ﬁle descriptor less than 256 very small. To solve this problem, the TSfopen family can open ﬁles with descriptors greater than 256. The TSfopen family of routines is not intended for high speed IO or ﬂexibility - they are blocking APIs (not asyn- chronous). For performance reasons, you should not directly use these APIs on a Trafﬁc Server thread (when being called back on an HTTP hook); it is better to use a separate thread for doing the blocking IO. The TSfopen family is intended for reading and writing conﬁguration information when corresponding usage of the fopen family of func- tions is inappropriate due to ﬁle descriptor and portability limitations. The TSfopen family of functions consists of the following: • TSfclose() • TSfflush() • TSfgets() • TSfopen() • TSfread() • TSfwrite()
Memory Allocation
Trafﬁc Server provides ﬁve routines for allocating and freeing memory. These routines correspond to similar routines in the C library. For example, TSrealloc behaves like the C library routine realloc. There are two main reasons for using the routines provided by Trafﬁc Server. The ﬁrst is portability: the Trafﬁc Server API routines behave the same on all of Trafﬁc Server’s supported platforms. For example, realloc does not accept an argument of NULL on some platforms. The second reason is that the Trafﬁc Server routines actually track the memory allocations by ﬁle and line number. This tracking is very efﬁcient, always turned on, and quite useful when tracking down memory leaks. The memory allocation functions are: • TSfree() • TSmalloc() • TSrealloc() • TSstrdup() • TSstrndup()
Thread Functions
The Trafﬁc Server API thread functions enable you to create, destroy, and identify threads within Trafﬁc Server. Multithreading enables a single program to have more than one stream of execution and to process more than one transaction at a time. Threads serialize their access to shared resources and data using the TSMutex type, as described in Mutexes. The thread functions are listed below:
4.7. Plugin Development 363 Apache-Trafﬁcserver-Server Documentation, latest
• TSThreadCreate() • TSThreadDestroy() • TSThreadInit() • TSThreadSelf()
Debugging Functions
• TSDebug() prints out a formatted statement if you are running Trafﬁc Server in debug mode. • TSIsDebugTagSet() checks to see if a debug tag is set. If the debug tag is set, then Trafﬁc Server prints out all debug statements associated with the tag. • TSError() prints error messages to Trafﬁc Server’s error log • TSAssert() enables the use of assertion in a plugin. • TSReleaseAssert() enables the use of assertion in a plugin.
Adding Statistics
This chapter describes how to add statistics to your plugins. Statistics can be coupled or uncoupled. Coupled statistics are quantities that are related and must therefore be updated together. The Trafﬁc Server API statistics functions add your plugin’s statistics to the Trafﬁc Server statistics system. You can view your plugin statistics as you would any other Trafﬁc Server statistic, using Trafﬁc Line (Trafﬁc Server’s command line interface). This chapter contains the following topics:
Uncoupled Statistics
A statistic is an object of type TSStat. The value of the statistic is of type TSStatType. The possible TSStatTypes are: • TSSTAT_TYPE_INT64 • TSSTAT_TYPE_FLOAT There is no TSSTAT_TYPE_INT32. To add uncoupled statistics, follow the steps below: 1. Declare your statistic as a global variable in your plugin. For example:
 static TSStat my_statistic;
2. In TSPluginInit, create new statistics using TSStatCreate. When you create a new statistic, you need to give it an “external” name that the Trafﬁc Server command line interface (Trafﬁc Line) uses to access the statistic. For example:
 my_statistic= TSStatCreate ("my.statistic", TSSTAT_TYPE_INT64);
3. Modify (increment, decrement, or other modiﬁcation) your statistic in plugin functions.
364 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Coupled Statistics
Use coupled statistics for quantities that are related and therefore must be updated jointly. As a very simple example, suppose you have three statistics: sum, part_1, and part_2. They must always preserve the relationship that sum = part_1 + part_2. If you update part_1 without updating sum at the same time, then the equation becomes untrue. Therefore, the statistics are said to be coupled. The mechanism for updating coupled statistics jointly is to create local copies of global coupled statistics in the routines that modiﬁy them. When each local copy is updated appropriately, do a global update using TSStatsCoupledUpdate. To specify which statistics are related to one another, establish a coupled statistic category and make sure that each coupled statistic belongs to the appropriate category. When it is time to do the global update, specify the category to be updated.
: The local statistic copy must have a duplicate set of statistics as that of the master copy. Local statistics must also be added to the local statistic category in the same order as their master copy counterparts were originally added.
Below are the steps you need to follow, along with a code example taken from the redirect-1.c sample plugin.
To add coupled statistics
1. Declare the global category for your coupled statistics as a global TSCoupledStat variable in your plugin. 2. Declare your coupled statistics as global TSStat variables in your plugin. 3. In TSPluginInit, create a new global coupled category using TSStatCoupledGlobalCategoryCreate. 4. In TSPluginInit, create new global coupled statistics using TSStatCoupledGlobalAdd. When you create a new statistic, you need to give it an “external” name that the Trafﬁc Server command line interface (Trafﬁc Line) uses to access the statistic. 5. In any routine wherein you want to modify (increment, decrement, or other modiﬁcation) your coupled statistics, declare local copies of the coupled category and coupled statistics. 6. Create local copies using TSStatCoupledLocalCopyCreate and TSStatCoupledLocalAdd. 7. Modify the local copies of your statistics. Then call TSStatsCoupledUpdate to update the global copies jointly. 8. When you are ﬁnished, you must destroy all of the local copies in the category via TSStatCoupledLocalCopyDestroy.
Example Using the redirect-1.c Sample Plugin static TSCoupledStat request_outcomes; static TSStat requests_all; static TSStat requests_redirects; static TSStat requests_unchanged; request_outcomes= TSStatCoupledGlobalCategoryCreate ("request_outcomes"); requests_all= TSStatCoupledGlobalAdd (request_outcomes,"requests.all", TSSTAT_TYPE_
˓→FLOAT); requests_redirects= TSStatCoupledGlobalAdd (request_outcomes,"requests.redirects", TSSTAT_TYPE_INT64);
4.7. Plugin Development 365 Apache-Trafﬁcserver-Server Documentation, latest
 requests_unchanged= TSStatCoupledGlobalAdd (request_outcomes,"requests.unchanged", TSSTAT_TYPE_INT64);
TSCoupledStat local_request_outcomes; TSStat local_requests_all; TSStat local_requests_redirects; TSStat local_requests_unchanged; local_request_outcomes= TSStatCoupledLocalCopyCreate("local_request_outcomes", request_outcomes); local_requests_all= TSStatCoupledLocalAdd(local_request_outcomes,"requests.all.local
˓→", TSSTAT_TYPE_FLOAT); local_requests_redirects= TSStatCoupledLocalAdd(local_request_outcomes, "requests.redirects.local", TSSTAT_TYPE_INT64); local_requests_unchanged= TSStatCoupledLocalAdd(local_request_outcomes, "requests.unchanged.local", TSSTAT_TYPE_INT64);
TSStatFloatAddTo( local_requests_all, 1.0); ... TSStatIncrement (local_requests_unchanged); TSStatsCoupledUpdate(local_request_outcomes);
TSStatCoupledLocalCopyDestroy(local_request_outcomes);
Viewing Statistics Using Trafﬁc Line
To view statistics for your plugin, follow the steps below: 1. Make sure you know the name of your statistic (i.e., the name used in the TSStatCoupledGlobalAdd, TSStatCreate, or TSStatCoupledGlobalCategoryCreate call). 2. In your /bin directory, enter the following:
./traffic\_line-r the\_name
Example Plugins
Basic Authorization Plugin
The sample basic authorization plugin, basic-auth.c, checks for basic HTTP proxy authorization. In HTTP basic proxy authorization, client user names and passwords are contained in the Proxy-Authorization header. The password is encoded using base64 encoding. The plugin checks all incoming requests for the authorization header, user name, and password. If the plugin does not ﬁnd all of the these, then it reenables with an error (effectively stopping the transaction) and adds a transaction hook to the send response header event.
Creating the Plugin’s Parent Continuation and Global Hook
The parent continuation and global hook are created as follows: TSHttpHookAdd (TS_HTTP_OS_DNS_HOOK, TSContCreate (auth_plugin, NULL));
366 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
: Authorization plugins which attach to TS_HTTP_OS_DNS_HOOK (as shown in the example above) will not operate as expected unless :ts:cv:‘proxy.conﬁg.http.doc_in_cache_skip_dns‘ is set to 0. Disabling this feature ensures that DNS hooks will still be executed even when a matching document has been located in the cache. The downside is that the performance gain by skipping otherwise unnecessary DNS lookups is lost.
Implementing the Handler and Getting a Handle to the Transaction
The handler function for the plugin’s parent continuation is implemented as follows:
 static int auth_plugin (TSCont contp, TSEvent event, void *edata) {
TSHttpTxn txnp= (TSHttpTxn) edata; switch (event) { case TS_EVENT_HTTP_OS_DNS: handle_dns (txnp, contp); return 0; case TS_EVENT_HTTP_SEND_RESPONSE_HDR: handle_response (txnp); return 0; default: break; }
 return 0; }
Working With HTTP Headers
The plugin checks all client request headers for the Proxy-Authorization MIME ﬁeld, which should contain the user name and password. The plugin’s continuation handler, auth-plugin, calls handle_dns to check the Proxy-Authorization ﬁeld. The handle_dns routine uses TSHttpTxnClientReqGet and TSMimeHdrFieldFind to obtain the Proxy-Authorization ﬁeld:
{ TSMBuffer bufp; TSMLoc hdr_loc; TSMLoc field_loc; const char *val; char *user, *password;
 if (!TSHttpTxnClientReqGet (txnp,&bufp,&hdr_loc)) { TSError ("[basic_authorization] Couldn't retrieve client request header"); goto done; }
 field_loc= TSMimeHdrFieldFind (bufp, hdr_loc, TS_MIME_FIELD_PROXY_AUTHORIZATION);
If the Proxy-Authorization ﬁeld is present, then the plugin checks that the authentication type is “Basic”, and the user name and password are present and valid:
4.7. Plugin Development 367 Apache-Trafﬁcserver-Server Documentation, latest
 val= TSMimeHdrFieldValueStringGet (bufp, hdr_loc, field_loc,-1,&authval_length); if (!val) { TSError ("[basic_authorization] No value in Proxy-Authorization field"); TSHandleMLocRelease (bufp, hdr_loc, field_loc); TSHandleMLocRelease (bufp, TS_NULL_MLOC, hdr_loc); goto done; } if (strncmp (val,"Basic",5)!=0){ TSError ("[basic_authorization] No Basic auth type in Proxy-Authorization"); TSHandleMLocRelease (bufp, hdr_loc, field_loc); TSHandleMLocRelease (bufp, TS_NULL_MLOC, hdr_loc); goto done; } val+=5; while ((*val=='')||( *val=='\t')) { val+=1; } user= base64_decode (val); password= strchr (user,':'); if (!password) { TSError ("[basic_authorization] No password in authorization information"); TSfree (user); TSHandleMLocRelease (bufp, hdr_loc, field_loc); TSHandleMLocRelease (bufp, TS_NULL_MLOC, hdr_loc); goto done; } *password='\0'; password+=1; if (!authorized (user, password)) { TSError ("[basic_authorization] %s:%s not authorized", user, password); TSfree (user); TSHandleMLocRelease (bufp, hdr_loc, field_loc); TSHandleMLocRelease (bufp, TS_NULL_MLOC, hdr_loc); goto done; }
TSfree (user); TSHandleMLocRelease (bufp, hdr_loc, field_loc); TSHandleMLocRelease (bufp, TS_NULL_MLOC, hdr_loc); TSHttpTxnReenable (txnp, TS_EVENT_HTTP_CONTINUE); return;
Setting a Transaction Hook
If the request does not have the Proxy-Authorization ﬁeld set to Basic authorization or a valid user- name/password, then the plugin sends the 407 Proxy authorization required status code back to the client. The client will then prompt the user for a username and password, and then resend the request. In the handle_dns routine, the following lines handle the authorization error case: done: TSHttpTxnHookAdd (txnp, TS_HTTP_SEND_RESPONSE_HDR_HOOK, contp);
368 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSHttpTxnReenable (txnp, TS_EVENT_HTTP_ERROR);
If handle_dns does not ﬁnd the Proxy-Authorization ﬁeld set to Basic authorization or a valid user- name/password, then it adds a SEND_RESPONSE_HDR_HOOK to the transaction being processed. This means that Trafﬁc Server will call the plugin back when sending the client response. handle_dns reenables the transaction with TS_EVENT_HTTP_ERROR, which means that the plugin wants Trafﬁc Server to terminate the transaction. When Trafﬁc Server terminates the transaction, it sends the client an error message. Because of the SEND_RESPONSE_HDR_HOOK, Trafﬁc Server calls the plugin back. The auth-plugin routine calls handle_response to send the client a 407 status code. When the client resends the request with the Proxy-Authorization ﬁeld, a new transaction begins. handle_dns calls base64_decode to decode the username and password; handle_dns also calls authorized to validate the username and password. In this plugin, sample NT code is provided for password validation. UNIX programmers can supply their own validation mechanism.
Blacklist Plugin
The sample blacklisting plugin included in the Trafﬁc Server SDK is blacklist-1.c. This plugin checks every incoming HTTP client request against a list of blacklisted web sites. If the client requests a blacklisted site, then the plugin returns an Access forbidden message to the client. The ﬂow of HTTP processing with the blacklist plugin is illustrated in the ﬁgure titled Blacklist Plugin. This ex- ample also contains a simple conﬁguration management interface. It can read a list of blacklisted sites from a ﬁle (blacklist.txt) that can be updated by a Trafﬁc Server administrator. When the conﬁguration ﬁle is updated, Trafﬁc Server sends an event to the plugin that wakes it up to do some work.
Creating the Parent Continuation
You create the static parent continuation in the mandatory TSPluginInit function. This parent continuation ef- fectively is the plugin: the plugin executes only when this continuation receives an event from Trafﬁc Server. Trafﬁc Server passes the event as an argument to the continuation’s handler function. When you create continuations, you must create and specify their handler functions. You can specify an optional mutex lock when you create continuations. The mutex lock protects data shared by asynchronous processes. Because Trafﬁc Server has a multi-threaded design, race conditions can occur if several threads try to access the same continuation’s data. Here is how the static parent continuation is created in blacklist-1.c:
 void TSPluginInit( int argc, const char *argv[]) { // ... TSCont contp;
 contp= TSContCreate (blacklist_plugin, NULL); // ... }
The handler function for the plugin is blacklist_plugin, and the mutex is null. The continuation handler func- tion’s job is to handle the events that are sent to it; accordingly, the blacklist_plugin routine consists of a switch statement that covers each of the events that might be sent to it:
4.7. Plugin Development 369 Apache-Trafﬁcserver-Server Documentation, latest
 static int blacklist_plugin (TSCont contp, TSEvent event, void *edata) { TSHttpTxn txnp= (TSHttpTxn) edata; switch (event) { case TS_EVENT_HTTP_OS_DNS: handle_dns (txnp, contp); return 0; case TS_EVENT_HTTP_SEND_RESPONSE_HDR: handle_response (txnp); return 0; default: TSDebug ("blacklist_plugin","This event was unexpected: %d \n", ); break; } return 0; }
When you write handler functions, you have to anticipate any events that might be sent to the handler by hooks or by other functions. In the Blacklist plugin, TS_EVENT_OS_DNS is sent because of the global hook established in TSPluginInit, TS_EVENT_HTTP_SEND_RESPONSE_HDR is sent because the plugin contains a transaction hook (see Setting Up a Transaction Hook). It is good practice to have a default case in your switch statements.
Setting a Global Hook
Global hooks are always added in TSPluginInit using TSHttpHookAdd. The two arguments of TSHttpHookAdd are the hook ID and the continuation to call when processing the event corresponding to the hook. In blacklist-1.c, the global hook is added as follows:
TSHttpHookAdd (TS_HTTP_OS_DNS_HOOK, contp);
Above, TS_HTTP_OS_DNS_HOOK is the ID for the origin server DNS lookup hook and contp is the parent contin- uation created earlier. This means that the Blacklist plugin is called at every origin server DNS lookup. When it is called, the handler functio blacklist_plugin receives TS_EVENT_HTTP_OS_DNS and calls handle_dns to see if the request is forbidden.
Accessing the Transaction Being Processed
A continuation’s handler function is of type TSEventFunc; the prototype is as follows: static int function_name (TSCont contp, TSEvent event, void *edata) In general, the return value of the handler function is not used. The continuation argument is the continuation being called back, the event is the event being sent to the continuation, and the data pointed to by void *edata depends on the type of event. The data types for each event type are listed in Writing Handler Functions The key here is that if the event is an HTTP transaction event, then the data passed to the continuation’s handler is of type TSHttpTxn (a data type that represents HTTP transactions). Your plugin can then do things with the transaction. Here’s how it looks in the code for the Blacklist plugin’s handler:
 static int blacklist_plugin (TSCont contp, TSEvent event, void *edata) {
370 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSHttpTxn txnp= (TSHttpTxn) edata; switch (event) { case TS_EVENT_HTTP_OS_DNS: handle_dns (txnp, contp); return 0; case TS_EVENT_HTTP_SEND_RESPONSE_HDR: handle_response (txnp); return 0; default: break; } return 0; }
For example: when the origin server DNS lookup event is sent, blacklist_plugin can call handle_dnsand pass txnp as an argument.
Setting Up a Transaction Hook
The Blacklist plugin sends “access forbidden” messages to clients if their requests are directed to blacklisted hosts. Therefore, the plugin needs a transaction hook so it will be called back when Trafﬁc Server’s HTTP state machine reaches the “send response header” event. In the Blacklist plugin’s handle_dns routine, the transaction hook is added as follows:
TSMutexLock (sites_mutex); for (i=0; i< nsites; i++){ if (strncmp (host, sites[i], host_length)==0){ printf ("blacklisting site: %s\n", sites[i]); TSHttpTxnHookAdd (txnp, TS_HTTP_SEND_RESPONSE_HDR_HOOK, contp); TSHandleMLocRelease (bufp, hdr_loc, url_loc); TSHandleMLocRelease (bufp, TS_NULL_MLOC, hdr_loc); TSHttpTxnReenable (txnp, TS_EVENT_HTTP_ERROR); TSMutexUnlock (sites_mutex); return; } } TSMutexUnlock (sites_mutex); done: TSHttpTxnReenable (txnp, TS_EVENT_HTTP_CONTINUE);
This code fragment shows some interesting features. The plugin is comparing the requested site to the list of blacklisted sites. While the plugin is using the blacklist, it must acquire the mutex lock for the blacklist to prevent conﬁguration changes in the middle of a blacklisting operation. If the requested site is blacklisted, then the following things happen: 1. A transaction hook is added with TSHttpTxnHookAdd; the plugin is called back at the “send response header” event (i.e., the plugin sends an Access forbidden message to the client). You can see that in order to add a transaction hook, you need a handle to the transaction being processed. 2. The transaction is reenabled using TSHttpTxnReenable with TS_EVENT_HTTP_ERROR as its event argu- ment. Reenabling with an error event tells the HTTP state machine to stop the transaction and jump to the “send response header” state. Notice that if the requested site is not blacklisted, then the transaction is reenabled with the TS_EVENT_HTTP_CONTINUE event. 3. The string and TSMLoc data stored in the marshal buffer bufp is released by TSHandleMLocRelease (see Marshal Buffers). Release these handles before re-enabling the transaction.
4.7. Plugin Development 371 Apache-Trafﬁcserver-Server Documentation, latest
In general, whenever the plugin is doing something to a transaction, it must reenable the transaction when it is ﬁnished. In other words: every time your handler function handles a transaction event, it must call TSHttpTxnReenable when it is ﬁnished. Similarly, after your plugin handles session events (TS_EVENT_HTTP_SSN_START and TS_EVENT_HTTP_SSN_CLOSE), it must reenable the session with TSHttpSsnReenable. Reenabling the trans- action twice in the same plugin routine is a bad error.
Working with HTTP Header Functions
The Blacklist plugin examines the host header in every client transaction. This is done in the handle_dns routine, using TSHttpTxnClientReqGet, TSHttpHdrUrlGet, and TSUrlHostGet.
 static void handle_dns (TSHttpTxn txnp, TSCont contp) { TSMBuffer bufp; TSMLoc hdr_loc; TSMLoc url_loc; const char *host; int i; int host_length; if (TSHttpTxnClientReqGet(txnp,&bufp,&hdr_loc)!= TS_SUCCESS) { TSError("[blacklist] Couldn't retrieve client request header"); goto done; } if (TSHttpHdrUrlGet(bufp, hdr_loc,&url_loc)!= TS_SUCCESS) { TSError("[blacklist] Couldn't retrieve request url"); TSHandleMLocRelease(bufp, TS_NULL_MLOC, hdr_loc); goto done; } host= TSUrlHostGet(bufp, url_loc,&host_length); if (!host) { TSError("[blacklist] couldn't retrieve request hostname"); TSHandleMLocRelease(bufp, hdr_loc, url_loc); TSHandleMLocRelease(bufp, TS_NULL_MLOC, hdr_loc); goto done; }
To access the host header, the plugin must ﬁrst get the client request, retrieve the URL portion, and then obtain the host header. See HTTP Headers for more information about these calls. See Marshal Buffers for guidelines on using TSHandleMLocRelease.
Sample Source Code blacklist-1.c
The sample blacklisting plugin included in the Trafﬁc Server SDK is blacklist-1.c. This plugin checks every incoming HTTP client request against a list of blacklisted web sites. If the client requests a blacklisted site, then the plugin returns an Access forbidden message to the client. This plugin illustrates: • An HTTP transaction extension
372 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
• How to examine HTTP request headers • How to use the logging interface • How to use the plugin conﬁguration management interface
Query Remap Plugin
Example: Query Remap Plugin
The sample remap plugin, query_remap.c, maps client requests to a number of servers based on a hash of the request’s URL query parameter. This can be useful for spreading load for a given type of request among backend servers, while still maintaining “stickiness” to a single server for similar requests. For example, a search engine may want to send repeated queries for the same keywords to a server that has likely cached the result from a prior query.
Conﬁguration of query_remap
The query remap plugin will allow the query parameter name to be speciﬁed, along with the hostnames of the servers to hash across. Sample remap.config rules using query_remap will look like: map http://www.example.com/search http://srch1.example.com/search @plugin=query_remap.
˓→so @pparam=q @pparam=srch1.example.com @pparam=srch2.example.com @pparam=srch3.
˓→example.com map http://www.example.com/profiles http://prof1.example.com/profiles @plugin=query_
˓→remap.so @pparam=user_id @pparam=prof1.example.com @pparam=prof2.example.com
The ﬁrst @pparam speciﬁes the query param key for which the value will be hashed. The remaining parameters list the hostnames of the servers. A request for http://www.example.com/search?q=apache will match the ﬁrst rule. The plugin will look for the ‘‘q‘‘ parameter and hash the value ‘apache‘ to pick from among srch_[1-3]_. example.com to send the request. If the request does not include a ‘‘q‘‘ query parameter and the plugin decides not to modify the request, the default toURL ‘http://srch1.example.com/search‘ will be used by TS. The parameters are passed to the plugin’s tsremap_new_instance function. In query_remap, tsremap_new_instance creates a plugin-deﬁned query_remap_info struct to store its conﬁguration pa- rameters. The ihandle, an opaque pointer that can be used to pass per-instance data, is set to this struct pointer and will be passed to the tsremap_remap function when it is triggered for a request. typedef struct _query_remap_info { char *param_name; size_t param_len; char **hosts; int num_hosts; } query_remap_info; int tsremap_new_instance(int argc,char *argv[],ihandle *ih,char *errbuf,int errbuf_ ˓→size) { int i;
 if (argc param_name= strdup(argv[2]); qri->param_len= strlen(qri->param_name); qri->num_hosts= argc-3; qri->hosts=( char**) TSmalloc(qri->num_hosts*sizeof(char*));
4.7. Plugin Development 373 Apache-Trafﬁcserver-Server Documentation, latest
 for (i=0; i num_hosts;++i) { qri->hosts[i]= strdup(argv[i+3]); }
*ih= (ihandle)qri; return 0; }
Another way remap plugins may want handle more complex conﬁguration is to specify a conﬁguration ﬁlename as a pparam and parse the speciﬁed ﬁle during instance initialization.
Performing the Remap
The plugin implements the tsremap_remap function, which is called when TS has read the client HTTP request headers and matched the request to a remap rule conﬁgured for the plugin. The TSRemapRequestInfo struct contains input and output members for the remap operation. tsremap_remap uses the conﬁguration information passed via the ihandle and checks the request_query for the conﬁgured query parameter. If the parameter is found, the plugin sets a new_host to modify the request host: int tsremap_remap(ihandle ih, rhandle rh, TSRemapRequestInfo *rri) { int hostidx=-1; query_remap_info *qri= (query_remap_info *)ih;
 if (!qri) { TSError("[remap] NULL ihandle"); return 0; }
 if (rri&& rri->request_query&& rri->request_query_size>0){ char *q, *s, *key;
//make a copy of the query, as it is read only q=( char*) TSmalloc(rri->request_query_size+1); strncpy(q, rri->request_query, rri->request_query_size); q[rri->request_query_size]='\0';
 s= q; //parse query parameters for (key= strsep(&s,"&"); key!= NULL; key= strsep(&s,"&")) { char *val= strchr(key,'='); if (val&&( size_t)(val-key)== qri->param_len&& !strncmp(key, qri->param_name, qri->param_len)) { ++val; //the param key matched the configured param_name //hash the param value to pick a host hostidx= hash_fnv32(val, strlen(val))%( uint32_t)qri->num_hosts; break; } }
TSfree(q);
 if (hostidx>=0){ rri->new_host_size= strlen(qri->hosts[hostidx]);
374 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 if (rri->new_host_size new_host, qri->hosts[hostidx], rri->new_host_size); return 1; //host has been modified } } }
//the request was not modified, TS will use the toURL from the remap rule return 0; }
The Remap plugin provides a more ﬂexible, dynamic way of specifying remap rules. It is not built on top of the Trafﬁc Server APIs and exists solely for the purpose of URL remapping. The remap plugin is not global –it is conﬁgured on a per-remap rule basis, which enables you to customize how URLs are redirected based on individual rules in the remap.config ﬁle. The Trafﬁc Server Remap API enables a plugin to dynamically map a client request to a target URL. Each plugin is associated with one or more remap rules in remap.config (an “instance”). If a request URL matches a remap rule’s “fromURL”, then Trafﬁc Server calls the plugin-deﬁned remap function for that request. ((Editor’s note: additional text TBD; text in this chapter is still under development))
Remap Header File
The remap.h header ﬁle contains the Trafﬁc Server remap API. By default, the header ﬁle location is: /usr/ local/include/ts/remap.h
Required Functions
A remap plugin is required to implement the following functions: • TSRemapInit: the remap initialization function, called once when the plugin is loaded • TSRemapNewInstance: a new instance is created for each rule associated with the plugin. Called each time the plugin used in a remap rule (this function is what processes the pparam values) • TSRemapDoRemap: the entry point used by Trafﬁc Server to ﬁnd the new URL to which it remaps; called every time a request comes in
Conﬁguration
To associate a remap plugin with a remap rule, use the @plugin parameter. See the Admin Guide section (?TBD?) for details on conﬁguring remap plugins
Header-Based Plugin Examples
Header-based plugins read or modify the headers of HTTP messages that Trafﬁc Server sends and receives. Reading this chapter will help you to understand the following topics: • Creating continuations for your plugins • Adding global hooks • Adding transaction hooks • Working with HTTP header functions
4.7. Plugin Development 375 Apache-Trafﬁcserver-Server Documentation, latest
The two sample plugins discussed in this chapter are blacklist-1.c and basic-auth.c.
Overview
Header-based plugins take actions based on the contents of HTTP request or response headers. Examples include ﬁltering (on the basis of requested URL, source IP address, or other request header), user authentication, or user redirection. Header-based plugins have the following common elements: • The plugin has a static parent continuation that scans all Trafﬁc Server headers (either request headers, response headers, or both). • The plugin has a global hook. This enables the plugin to check all transactions to determine if the plugin needs to do something. • The plugin gets a handle to the transaction being processed through the global hook. • If the plugin needs to do something to transactions in speciﬁc cases, then it sets up a transaction hook for a particular event. • The plugin obtains client header information and does something based on that information. This chapter demonstrates how these components are implemented in SDK sample code.
Conﬁguration Variable Implementation
Adding a new conﬁguration variable in records.config requires a number of steps which are mostly documented here. Before adding a new conﬁguration variable, please discuss it on the mailing list. It will commonly be the case that a better name, or a more general approach to the problem which solves several different issues, may be suggested.
Deﬁning the Variable
To begin, the new conﬁguration variables must be added to RecordsConfig.cc. This contains a long array of conﬁguration variable records. The ﬁelds for each record are: type:RecT Type of record. The valid values are: RECT_NULL Undeﬁned record. RECT_CONFIG General conﬁguration variable. RECT_PROCESS Process related statistic. RECT_NODE Local statistic. RECT_CLUSTER Cluster statistic. RECT_LOCAL Conﬁguration variable that is explicitly not shared across a cluster. RECT_PLUGIN Plugin created statistic. In general, RECT_CONFIG should be used unless it is required that the value not be shared among members of a cluster, in which case RECT_LOCAL should be used. If you use RECT_LOCAL, you must also start the line with LOCAL instead of CONFIG and the name should use .local. instead of .config.. name:char const* The fully qualiﬁed name of the conﬁguration variable. Although there appears to be a hierar- chial naming scheme, that’s just a convention, and it is not actually used by the code. Nonetheless, new variables should adhere to the hierarchial scheme.
376 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest value_type:RecDataT The data type of the value. It should be one of RECD_INT, RECD_STRING, RECD_FLOAT as appropriate. default:char const* The default value for the variable. This is always a string regardless of the value_type. update:RecUpdateT Information about how the variable is updated. The valid values are: RECU_NULL Behavior is unknown or unspeciﬁed. RECU_DYNAMIC This can be updated via command line tools. RECD_RESTART_TS The traffic_server process must be restarted for a new value to take effect. RECD_RESTART_TM The traffic_manager process must be restarted for a new value to take ef- fect. RECD_RESTART_TC The traffic_cop process must be restarted for a new value to take effect. required:RecordRequiredType Effectively a boolean that speciﬁes if the record is required to be present, with RR_NULL meaning not required and RR_REQUIRED indicating that it is required. Given that using RR_REQUIRED would be a major incompatibility, RR_NULL is generally the better choice. check:RecCheckT Additional type checking. It is unclear if this is actually implemented. The valid values are: RECC_NULL No additional checking. RECC_STR Verify the value is a string. RECC_INT Verify the value is an integer. RECC_IP Verify the value is an IP address. Unknown if this checks for IPv6. pattern:char const* This provides a regular expressions (PCRE format) for validating the value, beyond the basic type validation performed by RecCheckT. This can be NULL if there is no regular expression to use. access:RecAccessT Access control. The valid values are: RECA_NULL The value is read / write. RECA_READ_ONLY The value is read only. RECA_NO_ACCESS No access to the value; only privileged level parts of ATS can access the value.
Variable Infrastructure
The primary effort in deﬁning a conﬁguration variable is handling updates, generally via traffic_ctl config reload. This is handled in a generic way, as described in the next section, or in a more specialized way (built on top of the generic mechanism) for HTTP related conﬁguration variables. This is only needed if the variable is marked as dynamically updateable (RECU_DYNAMIC) although HTTP conﬁguration variables should be dynamic if possible.
Documentation and Defaults
A conﬁguration variable should be documented in records.config. There are many examples in the ﬁle already that can be used for guidance. The general format is to use the tag
.. ts:cv:`variable.name.here`
The arguments to this are the same as for the conﬁguration ﬁle. The documentation generator will pick out key bits and use them to decorate the entry. In particular if a value is present it will be removed and used as the default value. You can attach some additional options to the variable. These are: reloadable The variable can be reloaded via command line on a running Trafﬁc Server.
4.8. Conﬁguration Variable Implementation 377 Apache-Trafﬁcserver-Server Documentation, latest
 metric Specify the units for the value. This is critical for variables that use unexpected or non-obvious metrics, such as minutes instead of seconds, or disk sectors instead of bytes. deprecated Mark a variable as deprecated.
Example
:ts:cv:‘custom.variable‘ reloadable metric minutes deprecated
If you need to refer to another conﬁguration variable in the documentation, you can use the form
:ts:cv:`the.full.name.of.the.variable`
This will display the name as a link to the full deﬁnition. In general, a new conﬁguration variable should not be present in the default records.config. If it is added, such defaults should be added to the ﬁle proxy/config/records.config.default.in. This is used to generate the default records.config. Just add the variable to the ﬁle in an appropriate place with a proper default as this will now override whatever default you put in the code for new installs.
Handling Updates
The simplest mechanism for handling updates is the REC_EstablishStaticConfigXXX family of functions. This mechanism will cause the value in the indicated instance to be updated in place when an update to records. config occurs. This is done asynchronously using atomic operations. Use of these variables must keep that in mind. If a variable requires additional handling when updated a callback can be registered which is called when the variable is updated. This is what the REC_EstablishStaticConfigXXX calls do internally with a callback that simply reads the new value and writes it to storage indicated by the call parameters. The functions used are the link_XXX static functions in RecCore.cc. To register a conﬁguration variable callback, call RecRegisterConfigUpdateCb with the arguments: char const* name The variable name. callback A function with the signature . The :arg:‘name‘ value passed is the same as the :arg:‘name‘ passed to the registration function as is the :arg:‘cookie‘ argument. The :arg:‘type‘ and :arg:‘data‘ are the new value for the variable. The return value is currently ignored. For future compatibility return REC_ERR_OKAY. void* cookie A value passed to the callback. This is only for the callback, the internals simply store it and pass it on. callback is called under lock so it should be quick and not block. If that is necessary a continuation should be scheduled to handle the required action.
: The callback occurs asynchronously. For HTTP variables as described in the next section, this is handled by the more specialized HTTP update mechanisms. Otherwise it is the implementor’s responsibility to avoid race conditions.
378 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
HTTP Conﬁguation Values
Variables used for HTTP processing should be declared as members of the HTTPConfigParams structure (but see Overridable Variables for further details) and use the specialized HTTP update mechanisms which handle synchro- nization and initialization issues. The conﬁguration logic maintains two copies of the HTTPConfigParams structure, the master copy and the current copy. The master copy is kept in the m_master member of the HttpConfig singleton. The current copy is kept in the ConﬁgProcessor. The goal is to provide a (somewhat) atomic update for conﬁguration variables which are loaded individually in to the master copy as updates are received and then bulk copied to a new instance which is then swapped in as the current copy. The HTTP state machine interacts with this mechanism to avoid race conditions. For each variable, a mapping between the variable name and the appropriate member in the master copy should be established between in the HTTPConfig::startup method. The HttpEstablishStaticConfigXXX functions should be used unless there is a strong, explicit reason to not do so. The HTTPConfig::reconfigure method handles the current copy of the HTTP conﬁguration variables. Logic should be added here to copy the value from the master copy to the current copy. Generally this will be a simple assignment. If there are dependencies between variables, those should be checked and enforced in this method.
Overridable Variables
HTTP related variables that are changeable per transaction are stored in the OverridableHttpConfigParams structure, an instance of which is the oride member of HTTPConfigParams and therefore the points in the previous section still apply. The only difference for that is the further .oride member speciﬁer in the structure references. The variable is required to be accessible from the transaction API. In addition to any custom API functions used to access the value, the following items are required for generic access: 1. Add a value to the TSOverridableConfigKey enumeration in apidefs.h.in. 2. Augment the TSHttpTxnConfigFind function to return this enumeration value when given the name of the conﬁguration variable. Be sure to count the charaters very carefully. 3. Augment the _conf_to_memberp function in InkAPI.cc to return a pointer to the appropriate member of OverridableHttpConfigParams and set the type if not a byte value. 4. Update the testing logic in InkAPITest.cc by adding the string name of the conﬁguration variable to the SDK_Overridable_Configs array. 5. Update the Lua plugin enumeration TSLuaOverridableConfigKey in ts_lua_http_config.c. 6. Update the documentation of TSHttpOverridableConﬁg in TSHttpOverridableConfig.en.rst.
API Reference
For an introduction to the Trafﬁc Server API, how to link it to your plugin, and the methods of passing parameters to your plugin at runetime, please refer to TSAPI().
API Types
This section documents all of the types deﬁned by the Trafﬁc Server C API. While each type’s documentation refer- ences the C header ﬁle in which they are deﬁned, ts/apidefs.h, please note that your plugins should never need to include this ﬁle directly. Including the ts/ts.h header is sufﬁcient.
4.9. API Reference 379 Apache-Trafﬁcserver-Server Documentation, latest
TSCacheDataType
Synopsis
#include  TSCacheDataType Enum typedef.
Enumeration Members
TSCacheDataType TS_CACHE_DATA_TYPE_NONE TSCacheDataType TS_CACHE_DATA_TYPE_HTTP TSCacheDataType TS_CACHE_DATA_TYPE_OTHER
Description
TSCacheError
Synopsis
#include  TSCacheError Enum typedef.
Enumeration Members
TSCacheError TS_CACHE_ERROR_NO_DOC TSCacheError TS_CACHE_ERROR_DOC_BUSY TSCacheError TS_CACHE_ERROR_NOT_READY
Description
TSCacheLookupResult
Synopsis
#include  TSCacheLookupResult Enum typedef.
380 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Enumeration Members
TSCacheLookupResult TS_CACHE_LOOKUP_MISS TSCacheLookupResult TS_CACHE_LOOKUP_HIT_STALE TSCacheLookupResult TS_CACHE_LOOKUP_HIT_FRESH TSCacheLookupResult TS_CACHE_LOOKUP_SKIPPED
Description
TSCacheScanResult
Synopsis
#include  TSCacheScanResult Enum typedef.
Enumeration Members
TSCacheScanResult TS_CACHE_SCAN_RESULT_DONE TSCacheScanResult TS_CACHE_SCAN_RESULT_CONTINUE TSCacheScanResult TS_CACHE_SCAN_RESULT_DELETE TSCacheScanResult TS_CACHE_SCAN_RESULT_DELETE_ALL_ALTERNATES TSCacheScanResult TS_CACHE_SCAN_RESULT_UPDATE TSCacheScanResult TS_CACHE_SCAN_RESULT_RETRY
Description
TSEvent
Synopsis
#include  TSEvent Enum typedef deﬁning the possible events which may be passed to a continuation callback.
Enumeration Members
TSEvent TS_EVENT_NONE TSEvent TS_EVENT_IMMEDIATE TSEvent TS_EVENT_TIMEOUT
4.9. API Reference 381 Apache-Trafﬁcserver-Server Documentation, latest
TSEvent TS_EVENT_ERROR TSEvent TS_EVENT_CONTINUE TSEvent TS_EVENT_VCONN_READ_READY TSEvent TS_EVENT_VCONN_WRITE_READY TSEvent TS_EVENT_VCONN_READ_COMPLETE TSEvent TS_EVENT_VCONN_WRITE_COMPLETE TSEvent TS_EVENT_VCONN_EOS TSEvent TS_EVENT_VCONN_INACTIVITY_TIMEOUT TSEvent TS_EVENT_VCONN_ACTIVE_TIMEOUT TSEvent TS_EVENT_NET_CONNECT TSEvent TS_EVENT_NET_CONNECT_FAILED TSEvent TS_EVENT_NET_ACCEPT TSEvent TS_EVENT_NET_ACCEPT_FAILED TSEvent TS_EVENT_INTERNAL_206 TSEvent TS_EVENT_INTERNAL_207 TSEvent TS_EVENT_INTERNAL_208 TSEvent TS_EVENT_INTERNAL_209 TSEvent TS_EVENT_INTERNAL_210 TSEvent TS_EVENT_INTERNAL_211 TSEvent TS_EVENT_INTERNAL_212 TSEvent TS_EVENT_HOST_LOOKUP TSEvent TS_EVENT_CACHE_OPEN_READ TSEvent TS_EVENT_CACHE_OPEN_READ_FAILED TSEvent TS_EVENT_CACHE_OPEN_WRITE TSEvent TS_EVENT_CACHE_OPEN_WRITE_FAILED TSEvent TS_EVENT_CACHE_REMOVE TSEvent TS_EVENT_CACHE_REMOVE_FAILED TSEvent TS_EVENT_CACHE_SCAN TSEvent TS_EVENT_CACHE_SCAN_FAILED TSEvent TS_EVENT_CACHE_SCAN_OBJECT TSEvent TS_EVENT_CACHE_SCAN_OPERATION_BLOCKED TSEvent TS_EVENT_CACHE_SCAN_OPERATION_FAILED TSEvent TS_EVENT_CACHE_SCAN_DONE TSEvent TS_EVENT_CACHE_LOOKUP TSEvent TS_EVENT_CACHE_READ TSEvent TS_EVENT_CACHE_DELETE
382 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSEvent TS_EVENT_CACHE_WRITE TSEvent TS_EVENT_CACHE_WRITE_HEADER TSEvent TS_EVENT_CACHE_CLOSE TSEvent TS_EVENT_CACHE_LOOKUP_READY TSEvent TS_EVENT_CACHE_LOOKUP_COMPLETE TSEvent TS_EVENT_CACHE_READ_READY TSEvent TS_EVENT_CACHE_READ_COMPLETE TSEvent TS_EVENT_INTERNAL_1200 TSEvent TS_AIO_EVENT_DONE TSEvent TS_EVENT_HTTP_CONTINUE TSEvent TS_EVENT_HTTP_ERROR TSEvent TS_EVENT_HTTP_READ_REQUEST_HDR TSEvent TS_EVENT_HTTP_OS_DNS TSEvent TS_EVENT_HTTP_SEND_REQUEST_HDR TSEvent TS_EVENT_HTTP_READ_CACHE_HDR TSEvent TS_EVENT_HTTP_READ_RESPONSE_HDR TSEvent TS_EVENT_HTTP_SEND_RESPONSE_HDR TSEvent TS_EVENT_HTTP_REQUEST_TRANSFORM TSEvent TS_EVENT_HTTP_RESPONSE_TRANSFORM TSEvent TS_EVENT_HTTP_SELECT_ALT TSEvent TS_EVENT_HTTP_TXN_START TSEvent TS_EVENT_HTTP_TXN_CLOSE TSEvent TS_EVENT_HTTP_SSN_START TSEvent TS_EVENT_HTTP_SSN_CLOSE TSEvent TS_EVENT_HTTP_CACHE_LOOKUP_COMPLETE TSEvent TS_EVENT_HTTP_PRE_REMAP TSEvent TS_EVENT_HTTP_POST_REMAP TSEvent TS_EVENT_LIFECYCLE_PORTS_INITIALIZED TSEvent TS_EVENT_LIFECYCLE_PORTS_READY TSEvent TS_EVENT_LIFECYCLE_CACHE_READY TSEvent TS_EVENT_LIFECYCLE_SERVER_SSL_CTX_INITIALIZED TSEvent TS_EVENT_LIFECYCLE_CLIENT_SSL_CTX_INITIALIZED TSEvent TS_EVENT_VCONN_PRE_ACCEPT TSEvent TS_EVENT_MGMT_UPDATE TSEvent TS_EVENT_INTERNAL_60200 TSEvent TS_EVENT_INTERNAL_60201
4.9. API Reference 383 Apache-Trafﬁcserver-Server Documentation, latest
TSEvent TS_EVENT_INTERNAL_60202
Description
TSFetchWakeUpOptions
Synopsis
#include  TSFetchWakeUpOptions Enum typedef.
Enumeration Members
TSFetchWakeUpOptions NO_CALLBACK TSFetchWakeUpOptions AFTER_HEADER TSFetchWakeUpOptions AFTER_BODY
Description
TSHttpHookID
Synopsis
#include  TSHttpHookID Enum typedef deﬁning the possible Hooks for setting up Continuations callbacks.
Enumeration Members
TSHttpHookID TS_HTTP_READ_REQUEST_HDR_HOOK TSHttpHookID TS_HTTP_OS_DNS_HOOK TSHttpHookID TS_HTTP_SEND_REQUEST_HDR_HOOK TSHttpHookID TS_HTTP_READ_CACHE_HDR_HOOK TSHttpHookID TS_HTTP_READ_RESPONSE_HDR_HOOK TSHttpHookID TS_HTTP_SEND_RESPONSE_HDR_HOOK TSHttpHookID TS_HTTP_REQUEST_TRANSFORM_HOOK TSHttpHookID TS_HTTP_RESPONSE_TRANSFORM_HOOK TSHttpHookID TS_HTTP_SELECT_ALT_HOOK TSHttpHookID TS_HTTP_TXN_START_HOOK TSHttpHookID TS_HTTP_TXN_CLOSE_HOOK
384 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSHttpHookID TS_HTTP_SSN_START_HOOK TSHttpHookID TS_HTTP_SSN_CLOSE_HOOK TSHttpHookID TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK TSHttpHookID TS_HTTP_PRE_REMAP_HOOK TSHttpHookID TS_HTTP_POST_REMAP_HOOK TSHttpHookID TS_HTTP_RESPONSE_CLIENT_HOOK TSHttpHookID TS_SSL_FIRST_HOOK TSHttpHookID TS_VCONN_PRE_ACCEPT_HOOK TSHttpHookID TS_SSL_SNI_HOOK TSHttpHookID TS_SSL_CERT_HOOK TSHttpHookID TS_SSL_LAST_HOOK TSHttpHookID TS_HTTP_LAST_HOOK
Description
TSHttpStatus
Synopsis
#include  TSHttpStatus Enum typedef which deﬁnes the possible return values from TSHttpStatusGet().
Enumeration Members
TSHttpStatus TS_HTTP_STATUS_NONE TSHttpStatus TS_HTTP_STATUS_CONTINUE TSHttpStatus TS_HTTP_STATUS_SWITCHING_PROTOCOL TSHttpStatus TS_HTTP_STATUS_OK TSHttpStatus TS_HTTP_STATUS_CREATED TSHttpStatus TS_HTTP_STATUS_ACCEPTED TSHttpStatus TS_HTTP_STATUS_NON_AUTHORITATIVE_INFORMATION TSHttpStatus TS_HTTP_STATUS_NO_CONTENT TSHttpStatus TS_HTTP_STATUS_RESET_CONTENT TSHttpStatus TS_HTTP_STATUS_PARTIAL_CONTENT TSHttpStatus TS_HTTP_STATUS_MULTI_STATUS TSHttpStatus TS_HTTP_STATUS_ALREADY_REPORTED TSHttpStatus TS_HTTP_STATUS_IM_USED TSHttpStatus TS_HTTP_STATUS_MULTIPLE_CHOICES
4.9. API Reference 385 Apache-Trafﬁcserver-Server Documentation, latest
TSHttpStatus TS_HTTP_STATUS_MOVED_PERMANENTLY TSHttpStatus TS_HTTP_STATUS_MOVED_TEMPORARILY TSHttpStatus TS_HTTP_STATUS_SEE_OTHER TSHttpStatus TS_HTTP_STATUS_NOT_MODIFIED TSHttpStatus TS_HTTP_STATUS_USE_PROXY TSHttpStatus TS_HTTP_STATUS_TEMPORARY_REDIRECT TSHttpStatus TS_HTTP_STATUS_PERMANENT_REDIRECT TSHttpStatus TS_HTTP_STATUS_BAD_REQUEST TSHttpStatus TS_HTTP_STATUS_UNAUTHORIZED TSHttpStatus TS_HTTP_STATUS_PAYMENT_REQUIRED TSHttpStatus TS_HTTP_STATUS_FORBIDDEN TSHttpStatus TS_HTTP_STATUS_NOT_FOUND TSHttpStatus TS_HTTP_STATUS_METHOD_NOT_ALLOWED TSHttpStatus TS_HTTP_STATUS_NOT_ACCEPTABLE TSHttpStatus TS_HTTP_STATUS_PROXY_AUTHENTICATION_REQUIRED TSHttpStatus TS_HTTP_STATUS_REQUEST_TIMEOUT TSHttpStatus TS_HTTP_STATUS_CONFLICT TSHttpStatus TS_HTTP_STATUS_GONE TSHttpStatus TS_HTTP_STATUS_LENGTH_REQUIRED TSHttpStatus TS_HTTP_STATUS_PRECONDITION_FAILED TSHttpStatus TS_HTTP_STATUS_REQUEST_ENTITY_TOO_LARGE TSHttpStatus TS_HTTP_STATUS_REQUEST_URI_TOO_LONG TSHttpStatus TS_HTTP_STATUS_UNSUPPORTED_MEDIA_TYPE TSHttpStatus TS_HTTP_STATUS_REQUESTED_RANGE_NOT_SATISFIABLE TSHttpStatus TS_HTTP_STATUS_EXPECTATION_FAILED TSHttpStatus TS_HTTP_STATUS_UNPROCESSABLE_ENTITY TSHttpStatus TS_HTTP_STATUS_LOCKED TSHttpStatus TS_HTTP_STATUS_FAILED_DEPENDENCY TSHttpStatus TS_HTTP_STATUS_UPGRADE_REQUIRED TSHttpStatus TS_HTTP_STATUS_PRECONDITION_REQUIRED TSHttpStatus TS_HTTP_STATUS_TOO_MANY_REQUESTS TSHttpStatus TS_HTTP_STATUS_REQUEST_HEADER_FIELDS_TOO_LARGE TSHttpStatus TS_HTTP_STATUS_INTERNAL_SERVER_ERROR TSHttpStatus TS_HTTP_STATUS_NOT_IMPLEMENTED TSHttpStatus TS_HTTP_STATUS_BAD_GATEWAY TSHttpStatus TS_HTTP_STATUS_SERVICE_UNAVAILABLE
386 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSHttpStatus TS_HTTP_STATUS_GATEWAY_TIMEOUT TSHttpStatus TS_HTTP_STATUS_HTTPVER_NOT_SUPPORTED TSHttpStatus TS_HTTP_STATUS_VARIANT_ALSO_NEGOTIATES TSHttpStatus TS_HTTP_STATUS_INSUFFICIENT_STORAGE TSHttpStatus TS_HTTP_STATUS_LOOP_DETECTED TSHttpStatus TS_HTTP_STATUS_NOT_EXTENDED TSHttpStatus TS_HTTP_STATUS_NETWORK_AUTHENTICATION_REQUIRED
Description
TSHttpType
Synopsis
#include  TSHttpType Enum typedef which deﬁnes the possible HTTP types assigned to an HTTP header, as returned by TSHttpHdrTypeGet(). Headers created by TSHttpHdrCreate() receive TS_HTTP_TYPE_UNKNOWN by default and may be modiﬁed once by using TSHttpHdrTypeSet().
Enumeration Members
TSHttpType TS_HTTP_TYPE_UNKNOWN Default for new headers created by TSHttpHdrCreate(). TSHttpType TS_HTTP_TYPE_REQUEST HTTP request headers. TSHttpType TS_HTTP_TYPE_RESPONSE HTTP response headers.
Description
TSIOBuffersSizeIndex
Synopsis
#include  TSIOBuffersSizeIndex Enum typedef.
4.9. API Reference 387 Apache-Trafﬁcserver-Server Documentation, latest
Enumeration Members
TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_128 TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_256 TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_512 TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_1K TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_2K TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_4K TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_8K TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_16K TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_32K TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_64K TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_128K TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_256K TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_512K TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_1M TSIOBuffersSizeIndex TS_IOBUFFER_SIZE_INDEX_2M
Description
TSLifecycleHookID
Synopsis
#include  TSLifeCycleHookID Enum typedef used to indicate the event hook being called during a continuation invocation.
Enumeration Members
TSLifecycleHookID TS_LIFECYCLE_PORTS_INITIALIZED_HOOK TSLifecycleHookID TS_LIFECYCLE_PORTS_READY_HOOK TSLifecycleHookID TS_LIFECYCLE_CACHE_READY_HOOK TSLifecycleHookID TS_LIFECYCLE_SERVER_SSL_CTX_INITIALIZED_HOOK TSLifecycleHookID TS_LIFECYCLE_CLIENT_SSL_CTX_INITIALIZED_HOOK TSLifecycleHookID TS_LIFECYCLE_LAST_HOOK
388 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
TSLookingUpType
Synopsis
#include  TSLookingUpType Enum typedef.
Enumeration Members
TSLookingUpType TS_LOOKUP_UNDEFINED_LOOKUP TSLookingUpType TS_LOOKUP_ICP_SUGGESTED_HOST TSLookingUpType TS_LOOKUP_PARENT_PROXY TSLookingUpType TS_LOOKUP_ORIGIN_SERVER TSLookingUpType TS_LOOKUP_INCOMING_ROUTER TSLookingUpType TS_LOOKUP_HOST_NONE
Description
TSMiletonesType
Synopsis
#include  TSMiletonesType Enum typedef.
Enumeration Members
TSMiletonesType TS_MILESTONE_NULL TSMiletonesType TS_MILESTONE_UA_BEGIN TSMiletonesType TS_MILESTONE_UA_READ_HEADER_DONE TSMiletonesType TS_MILESTONE_UA_BEGIN_WRITE TSMiletonesType TS_MILESTONE_UA_CLOSE TSMiletonesType TS_MILESTONE_SERVER_FIRST_CONNECT TSMiletonesType TS_MILESTONE_SERVER_CONNECT TSMiletonesType TS_MILESTONE_SERVER_CONNECT_END TSMiletonesType TS_MILESTONE_SERVER_BEGIN_WRITE TSMiletonesType TS_MILESTONE_SERVER_FIRST_READ
4.9. API Reference 389 Apache-Trafﬁcserver-Server Documentation, latest
TSMiletonesType TS_MILESTONE_SERVER_READ_HEADER_DONE TSMiletonesType TS_MILESTONE_SERVER_CLOSE TSMiletonesType TS_MILESTONE_CACHE_OPEN_READ_BEGIN TSMiletonesType TS_MILESTONE_CACHE_OPEN_READ_END TSMiletonesType TS_MILESTONE_CACHE_OPEN_WRITE_BEGIN TSMiletonesType TS_MILESTONE_CACHE_OPEN_WRITE_END TSMiletonesType TS_MILESTONE_DNS_LOOKUP_BEGIN TSMiletonesType TS_MILESTONE_DNS_LOOKUP_END TSMiletonesType TS_MILESTONE_SM_START TSMiletonesType TS_MILESTONE_SM_FINISH TSMiletonesType TS_MILESTONE_LAST_ENTRY
Description
TSOverridableConﬁgKey
Synopsis
#include  TSOverridableConfigKey Enum typedef.
Enumeration Members
TSOverridableConﬁgKey TS_CONFIG_NULL TSOverridableConﬁgKey TS_CONFIG_URL_REMAP_PRISTINE_HOST_HDR TSOverridableConﬁgKey TS_CONFIG_HTTP_CHUNKING_ENABLED TSOverridableConﬁgKey TS_CONFIG_HTTP_NEGATIVE_CACHING_ENABLED TSOverridableConﬁgKey TS_CONFIG_HTTP_NEGATIVE_CACHING_LIFETIME TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_WHEN_TO_REVALIDATE TSOverridableConﬁgKey TS_CONFIG_HTTP_KEEP_ALIVE_ENABLED_IN TSOverridableConﬁgKey TS_CONFIG_HTTP_KEEP_ALIVE_ENABLED_OUT TSOverridableConﬁgKey TS_CONFIG_HTTP_KEEP_ALIVE_POST_OUT TSOverridableConﬁgKey TS_CONFIG_HTTP_SHARE_SERVER_SESSIONS, // DEPRECATED TSOverridableConﬁgKey TS_CONFIG_HTTP_SERVER_SESSION_SHARING_POOL TSOverridableConﬁgKey TS_CONFIG_HTTP_SERVER_SESSION_SHARING_MATCH TSOverridableConﬁgKey TS_CONFIG_NET_SOCK_RECV_BUFFER_SIZE_OUT TSOverridableConﬁgKey TS_CONFIG_NET_SOCK_SEND_BUFFER_SIZE_OUT TSOverridableConﬁgKey TS_CONFIG_NET_SOCK_OPTION_FLAG_OUT
390 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSOverridableConﬁgKey TS_CONFIG_HTTP_FORWARD_PROXY_AUTH_TO_PARENT TSOverridableConﬁgKey TS_CONFIG_HTTP_ANONYMIZE_REMOVE_FROM TSOverridableConﬁgKey TS_CONFIG_HTTP_ANONYMIZE_REMOVE_REFERER TSOverridableConﬁgKey TS_CONFIG_HTTP_ANONYMIZE_REMOVE_USER_AGENT TSOverridableConﬁgKey TS_CONFIG_HTTP_ANONYMIZE_REMOVE_COOKIE TSOverridableConﬁgKey TS_CONFIG_HTTP_ANONYMIZE_REMOVE_CLIENT_IP TSOverridableConﬁgKey TS_CONFIG_HTTP_ANONYMIZE_INSERT_CLIENT_IP TSOverridableConﬁgKey TS_CONFIG_HTTP_RESPONSE_SERVER_ENABLED TSOverridableConﬁgKey TS_CONFIG_HTTP_INSERT_SQUID_X_FORWARDED_FOR TSOverridableConﬁgKey TS_CONFIG_HTTP_SERVER_TCP_INIT_CWND TSOverridableConﬁgKey TS_CONFIG_HTTP_SEND_HTTP11_REQUESTS TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_HTTP TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_CLUSTER_CACHE_LOCAL TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_IGNORE_CLIENT_NO_CACHE TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_IGNORE_CLIENT_CC_MAX_AGE TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_IMS_ON_CLIENT_NO_CACHE TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_IGNORE_SERVER_NO_CACHE TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_CACHE_RESPONSES_TO_COOKIES TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_IGNORE_AUTHENTICATION TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_CACHE_URLS_THAT_LOOK_DYNAMIC TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_REQUIRED_HEADERS TSOverridableConﬁgKey TS_CONFIG_HTTP_INSERT_REQUEST_VIA_STR TSOverridableConﬁgKey TS_CONFIG_HTTP_INSERT_RESPONSE_VIA_STR TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_HEURISTIC_MIN_LIFETIME TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_HEURISTIC_MAX_LIFETIME TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_GUARANTEED_MIN_LIFETIME TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_GUARANTEED_MAX_LIFETIME TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_MAX_STALE_AGE TSOverridableConﬁgKey TS_CONFIG_HTTP_KEEP_ALIVE_NO_ACTIVITY_TIMEOUT_IN TSOverridableConﬁgKey TS_CONFIG_HTTP_KEEP_ALIVE_NO_ACTIVITY_TIMEOUT_OUT TSOverridableConﬁgKey TS_CONFIG_HTTP_TRANSACTION_NO_ACTIVITY_TIMEOUT_IN TSOverridableConﬁgKey TS_CONFIG_HTTP_TRANSACTION_NO_ACTIVITY_TIMEOUT_OUT TSOverridableConﬁgKey TS_CONFIG_HTTP_TRANSACTION_ACTIVE_TIMEOUT_OUT TSOverridableConﬁgKey TS_CONFIG_HTTP_ORIGIN_MAX_CONNECTIONS TSOverridableConﬁgKey TS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES TSOverridableConﬁgKey TS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_DEAD_SERVER
4.9. API Reference 391 Apache-Trafﬁcserver-Server Documentation, latest
TSOverridableConﬁgKey TS_CONFIG_HTTP_CONNECT_ATTEMPTS_RR_RETRIES TSOverridableConﬁgKey TS_CONFIG_HTTP_CONNECT_ATTEMPTS_TIMEOUT TSOverridableConﬁgKey TS_CONFIG_HTTP_POST_CONNECT_ATTEMPTS_TIMEOUT TSOverridableConﬁgKey TS_CONFIG_HTTP_DOWN_SERVER_CACHE_TIME TSOverridableConﬁgKey TS_CONFIG_HTTP_DOWN_SERVER_ABORT_THRESHOLD TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_FUZZ_TIME TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_FUZZ_MIN_TIME TSOverridableConﬁgKey TS_CONFIG_HTTP_DOC_IN_CACHE_SKIP_DNS TSOverridableConﬁgKey TS_CONFIG_HTTP_BACKGROUND_FILL_ACTIVE_TIMEOUT TSOverridableConﬁgKey TS_CONFIG_HTTP_RESPONSE_SERVER_STR TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_HEURISTIC_LM_FACTOR TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_FUZZ_PROBABILITY TSOverridableConﬁgKey TS_CONFIG_HTTP_BACKGROUND_FILL_COMPLETED_THRESHOLD TSOverridableConﬁgKey TS_CONFIG_NET_SOCK_PACKET_MARK_OUT TSOverridableConﬁgKey TS_CONFIG_NET_SOCK_PACKET_TOS_OUT TSOverridableConﬁgKey TS_CONFIG_HTTP_INSERT_AGE_IN_RESPONSE TSOverridableConﬁgKey TS_CONFIG_HTTP_CHUNKING_SIZE TSOverridableConﬁgKey TS_CONFIG_HTTP_FLOW_CONTROL_ENABLED TSOverridableConﬁgKey TS_CONFIG_HTTP_FLOW_CONTROL_LOW_WATER_MARK TSOverridableConﬁgKey TS_CONFIG_HTTP_FLOW_CONTROL_HIGH_WATER_MARK TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_RANGE_LOOKUP TSOverridableConﬁgKey TS_CONFIG_HTTP_NORMALIZE_AE_GZIP TSOverridableConﬁgKey TS_CONFIG_HTTP_DEFAULT_BUFFER_SIZE TSOverridableConﬁgKey TS_CONFIG_HTTP_DEFAULT_BUFFER_WATER_MARK TSOverridableConﬁgKey TS_CONFIG_HTTP_REQUEST_HEADER_MAX_SIZE TSOverridableConﬁgKey TS_CONFIG_HTTP_RESPONSE_HEADER_MAX_SIZE TSOverridableConﬁgKey TS_CONFIG_HTTP_NEGATIVE_REVALIDATING_ENABLED TSOverridableConﬁgKey TS_CONFIG_HTTP_NEGATIVE_REVALIDATING_LIFETIME TSOverridableConﬁgKey TS_CONFIG_HTTP_ACCEPT_ENCODING_FILTER_ENABLED TSOverridableConﬁgKey TS_CONFIG_SSL_HSTS_MAX_AGE TSOverridableConﬁgKey TS_CONFIG_SSL_HSTS_INCLUDE_SUBDOMAINS TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_OPEN_READ_RETRY_TIME TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_MAX_OPEN_READ_RETRIES TSOverridableConﬁgKey TS_CONFIG_HTTP_CACHE_RANGE_WRITE TSOverridableConﬁgKey TS_CONFIG_HTTP_POST_CHECK_CONTENT_LENGTH_ENABLED TSOverridableConﬁgKey TS_CONFIG_HTTP_GLOBAL_USER_AGENT_HEADER
392 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSOverridableConﬁgKey TS_CONFIG_LAST_ENTRY
Description
TSParseResult
Synopsis
#include  TSParseResult Enum typedef deﬁning the possible return values from TSHttpHdrParseResp().
Enumeration Members
TSParseResult TS_PARSE_ERROR TSParseResult TS_PARSE_DONE TSParseResult TS_PARSE_OK TSParseResult TS_PARSE_CONT
Description
TSRecordAccessType
Synopsis
#include  TSRecordAccessType Enum typedef.
Enumeration Members
TSRecordAccessType TS_RECORDACCESS_NULL TSRecordAccessType TS_RECORDACCESS_NO_ACCESS TSRecordAccessType TS_RECORDACCESS_READ_ONLY
Description
TSRecordCheckType
Synopsis
#include  TSRecordCheckType
4.9. API Reference 393 Apache-Trafﬁcserver-Server Documentation, latest
Enum typedef.
Enumeration Members
TSRecordCheckType TS_RECORDCHECK_NULL TSRecordCheckType TS_RECORDCHECK_STR TSRecordCheckType TS_RECORDCHECK_INT TSRecordCheckType TS_RECORDCHECK_IP
Description
TSRecordDataType
Synopsis
#include  TSRecordDataType Enum typedef.
Enumeration Members
TSRecordDataType TS_RECORDDATATYPE_NULL TSRecordDataType TS_RECORDDATATYPE_INT TSRecordDataType TS_RECORDDATATYPE_FLOAT TSRecordDataType TS_RECORDDATATYPE_STRING TSRecordDataType TS_RECORDDATATYPE_COUNTER TSRecordDataType TS_RECORDDATATYPE_STAT_CONST TSRecordDataType TS_RECORDDATATYPE_STAT_FX TSRecordDataType TS_RECORDDATATYPE_MAX
Description
TSRecordModeType
Synopsis
#include  TSRecordModeType Enum typedef.
394 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Enumeration Members
TSRecordModeType TS_RECORDMODE_NULL TSRecordModeType TS_RECORDMODE_CLIENT TSRecordModeType TS_RECORDMODE_SERVER TSRecordModeType TS_RECORDMODE_STAND_ALONE
Description
TSRecordPersistType
Synopsis
#include  TSRecordPersistType Enum typedef.
Enumeration Members
TSRecordPersistType TS_RECORDP_NULL TSRecordPersistType TS_RECORDP_PERSISTENT TSRecordPersistType TS_RECORDP_NON_PERSISTENT
Description
TSRecordType
Synopsis
#include  TSRecordType Enum typedef.
Enumeration Members
TSRecordType TS_RECORDTYPE_NULL TSRecordType TS_RECORDTYPE_CONFIG TSRecordType TS_RECORDTYPE_PROCESS TSRecordType TS_RECORDTYPE_NODE TSRecordType TS_RECORDTYPE_CLUSTER TSRecordType TS_RECORDTYPE_LOCAL
4.9. API Reference 395 Apache-Trafﬁcserver-Server Documentation, latest
TSRecordType TS_RECORDTYPE_PLUGIN TSRecordType TS_RECORDTYPE_ALL
Description
TSRecordUpdateType
Synopsis
#include  TSRecordUpdateType Enum typedef.
Enumeration Members
TSRecordUpdateType TS_RECORDUPDATE_NULL TSRecordUpdateType TS_RECORDUPDATE_DYNAMIC TSRecordUpdateType TS_RECORDUPDATE_RESTART_TS TSRecordUpdateType TS_RECORDUPDATE_RESTART_TM TSRecordUpdateType TS_RECORDUPDATE_RESTART_TC
Description
TSReturnCode
Synopsis
#include  TSReturnCode Enum typedef.
Enumeration Members
TSReturnCode TS_ERROR TSReturnCode TS_SUCCESS
Description
TSSDKVersion
Synopsis
#include 
396 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSSDKVersion Enum typedef.
Enumeration Members
TSSDKVersion TS_SDK_VERSION_2_0 TSSDKVersion TS_SDK_VERSION_3_0
Description
TSServerSessionSharingMatchType
Synopsis
#include  TSServerSessionSharingMatchType Enum typedef.
Enumeration Members
TSServerSessionSharingMatchType TS_SERVER_SESSION_SHARING_MATCH_NONE TSServerSessionSharingMatchType TS_SERVER_SESSION_SHARING_MATCH_BOTH TSServerSessionSharingMatchType TS_SERVER_SESSION_SHARING_MATCH_IP TSServerSessionSharingMatchType TS_SERVER_SESSION_SHARING_MATCH_HOST
Description
TSServerSessionSharingPoolType
Synopsis
#include  TSServerSessionSharingPoolType Enum typedef.
Enumeration Members
TSServerSessionSharingPoolType TS_SERVER_SESSION_SHARING_POOL_GLOBAL TSServerSessionSharingPoolType TS_SERVER_SESSION_SHARING_POOL_THREAD
4.9. API Reference 397 Apache-Trafﬁcserver-Server Documentation, latest
Description
TSServerState
Synopsis
#include  TSServerState Enum typedef.
Enumeration Members
TSServerState TS_SRVSTATE_STATE_UNDEFINED TSServerState TS_SRVSTATE_ACTIVE_TIMEOUT TSServerState TS_SRVSTATE_BAD_INCOMING_RESPONSE TSServerState TS_SRVSTATE_CONNECTION_ALIVE TSServerState TS_SRVSTATE_CONNECTION_CLOSED TSServerState TS_SRVSTATE_CONNECTION_ERROR TSServerState TS_SRVSTATE_INACTIVE_TIMEOUT TSServerState TS_SRVSTATE_OPEN_RAW_ERROR TSServerState TS_SRVSTATE_PARSE_ERROR TSServerState TS_SRVSTATE_TRANSACTION_COMPLETE TSServerState TS_SRVSTATE_CONGEST_CONTROL_CONGESTED_ON_F TSServerState TS_SRVSTATE_CONGEST_CONTROL_CONGESTED_ON_M
Description
TSSslVConnOp
Synopsis
#include  TSSslVConnOp Enum typedef.
Enumeration Members
TSSslVConnOp TS_SSL_HOOK_OP_DEFAULT TSSslVConnOp TS_SSL_HOOK_OP_TUNNEL TSSslVConnOp TS_SSL_HOOK_OP_TERMINATE TSSslVConnOp TS_SSL_HOOK_OP_LAST
398 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
TSThreadPool
Synopsis
#include  TSThreadPool Enum typedef.
Enumeration Members
TSThreadPool TS_THREAD_POOL_DEFAULT TSThreadPool TS_THREAD_POOL_NET TSThreadPool TS_THREAD_POOL_TASK TSThreadPool TS_THREAD_POOL_SSL TSThreadPool TS_THREAD_POOL_DNS TSThreadPool TS_THREAD_POOL_REMAP TSThreadPool TS_THREAD_POOL_CLUSTER TSThreadPool TS_THREAD_POOL_UDP
Description
TSVConnCloseFlags
Synopsis
#include  TSVConnCloseFlags Enum typedef.
Enumeration Members
TSVConnCloseFlags TS_VC_CLOSE_ABORT TSVConnCloseFlags TS_VC_CLOSE_NORMAL
Description
API Functions
TSAPI
Introduction to the Apache Trafﬁc Server API.
4.9. API Reference 399 Apache-Trafﬁcserver-Server Documentation, latest
Synopsis
#include  #include 
Description
The Apache Trafﬁc Server API enables you to create plugins, using the C programming language, that customize the behavior of your Trafﬁc Server installation. Trafﬁc Server enables sophisticated caching and processing of web-related trafﬁc, such as DNS and HTTP requests and responses. Trafﬁc Server itself consists of an event-driven loop that can be simpliﬁed as follows: for (;;) { event= get_next_event(); handle_event (event); }
You compile your plugin source code to create a shared library that Trafﬁc Server loads when it is started. Your plugin contains callback functions that are registered for speciﬁc Trafﬁc Server events. When Trafﬁc Server needs to process an event, it invokes any and all call-back functions you’ve registered for that event type. Possible uses for plugins include the following: • HTTP processing plugins can ﬁlter, blacklist, authorize users or transform content. • Protocol plugins can enable Trafﬁc Server to proxy-cache new protocol content. • A blacklisting plugin denies attempts to access web sites that are off-limits. • Append transform plugins add data to HTTP response content. • An image conversion plugin transforms JPEG images to GIF images. • Compression plugins send response content to a compression server that compresses the data (alternatively, a compression library local to the Trafﬁc Server host machine could do the compression). • Authorization plugins check a user’s permissions to access particular web sites. The plugin could consult a local authorization program or send queries to an authorization server. • A plugin that gathers client information from request headers and enters this information in a database. • A protocol plugin listen for speciﬁc protocol requests on a designated port and then uses Trafﬁc Server’s proxy server and cache to serve client requests.
Naming Conventions
The Trafﬁc Server API adheres to the following naming conventions: • The TS preﬁx is used for all function and variable names deﬁned in the Trafﬁc Server API. For example, TS_EVENT_NONE, TSMutex, and TSContCreate(). • Enumerated values are always written in all uppercase letters. For example, TS_EVENT_NONE and TS_VC_CLOSE_ABORT. • Constant values are all uppercase; enumerated values can be seen as a subset of constants. For example, TS_URL_SCHEME_FILE and TS_MIME_FIELD_ACCEPT. • The names of deﬁned types are mixed-case. For example, TSHttpSsn and TSHttpTxn(). TSDebug() • Function names are mixed-case. For example, TSUrlCreate() and TSContDestroy().
400 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
• Function names use the following subject-verb naming style: TS--, where  goes from general to speciﬁc. This makes it easier to determine what a function does by reading its name. For ex- ample, the function to retrieve the password ﬁeld (the speciﬁc subject) from a URL (the general subject) is TSUrlPasswordGet(). • Common verbs like Create, Destroy, Get, Set, Copy, Find, Retrieve, Insert, Remove, and Delete are used only when appropriate.
Plugin Loading and Conﬁguration
When Trafﬁc Server is ﬁrst started, it consults the plugin.conﬁg ﬁle to determine the names of all shared plugin libraries that need to be loaded. The plugin.conﬁg ﬁle also deﬁnes arguments that are to be passed to each plugin’s initialization function, TSPluginInit(). The records.config ﬁle deﬁnes the path to each plugin shared library. The sample plugin.config ﬁle below contains a comment line, a blank line, and two plugin conﬁgurations:
# This is a comment line. my-plugin.so www.junk.com www.trash.com www.garbage.com some-plugin.so arg1 arg2 $proxy.config.http.cache.on
Each plugin conﬁguration in the plugin.config ﬁle resembles a UNIX or DOS shell command; each line in plugin.config cannot exceed 1023 characters. The ﬁrst plugin conﬁguration is for a plugin named my-plugin.so. It contains three arguments that are to be passed to that plugin’s initialization routine. The second conﬁguration is for a plugin named some-plugin.so; it contains three arguments. The last argument, $proxy.conﬁg.http.cache.on, is actually a conﬁguration variable. Trafﬁc Server will look up the speciﬁed conﬁguration variable and substitute its value. Plugins are loaded and initialized by Trafﬁc Server in the order they appear in the plugin.config ﬁle.
Plugin Initialization
Each plugin must deﬁne an initialization function named TSPluginInit() that Trafﬁc Server invokes when the plugin is loaded. TSPluginInit() is commonly used to read conﬁguration information and register hooks for event notiﬁcation.
Files plugin.config, records.config
See Also
TSPluginInit(3ts)
TSActionCancel
Synopsis
#include  void TSActionCancel(TSAction actionp)
4.9. API Reference 401 Apache-Trafﬁcserver-Server Documentation, latest
Description
TSActionDone
Synopsis
#include  int TSActionDone(TSAction actionp)
Description
TSCacheRead
Synopsis
#include  TSAction TSCacheRead(TSCont contp, TSCacheKey key)
Description
Asks the Trafﬁc Server cache if the object corresponding to :arg:‘key‘ exists in the cache and can be read. If the object can be read, the Trafﬁc Server cache calls the continuation :arg:‘contp‘ back with the event TS_EVENT_CACHE_OPEN_READ. In this case, the cache also passes :arg:‘contp‘ a cache vconnection and :arg:‘contp‘ can then initiate a read operation on that vconnection using TSVConnRead. If the object cannot be read, the cache calls :arg:‘contp‘ back with the event TS_EVENT_CACHE_OPEN_READ_FAILED. The user (:arg:‘contp‘) has the option to cancel the action re- turned by TSCacheRead. Note that reentrant calls are possible, i.e. the cache can call back the user (:arg:‘contp‘) in the same call.
TSCacheRemove
Synopsis
#include  TSAction TSCacheRemove(TSCont contp, TSCacheKey key)
Description
Removes the object corresponding to :arg:‘key‘ from the cache. If the object was removed successfully, the cache calls :arg:‘contp‘ back with the event TS_EVENT_CACHE_REMOVE. If the object was not found in the cache, the cache calls :arg:‘contp‘ back with the event TS_EVENT_CACHE_REMOVE_FAILED. In both of these callbacks, the user (:arg:‘contp‘) does not have to do anything. The user does not get any vconnection from the cache, since no data needs to be transferred. When the cache calls :arg:‘contp‘ back with TS_EVENT_CACHE_REMOVE, the remove has already been commited.
402 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSCacheWrite
Synopsis
#include  TSAction TSCacheWrite(TSCont contp, TSCacheKey key)
Description
Asks the Trafﬁc Server cache if :arg:‘contp‘ can start writing the object corresponding to :arg:‘key‘ to the cache. If the object can be written, the cache calls :arg:‘contp‘ back with the event TS_EVENT_CACHE_OPEN_WRITE. In this case, the cache also passes :arg:‘contp‘ a cache vconnection and :arg:‘contp‘ can then initiate a write operation on that vconnection using TSVConnWrite. The object is not committed to the cache until the vconnection is closed. When all data has been transferred, the user (:arg:‘contp‘) must do an TSVConnClose. In case of any errors, the user must do an TSVConnAbort(contp, 0). If the object cannot be written, the cache calls :arg:‘contp‘ back with the event TS_EVENT_CACHE_OPEN_WRITE_FAILED. This can happen, for example, if there is another object with the same :arg:‘key‘ being written to the cache. The user (:arg:‘contp‘) has the option to cancel the action returned by TSCacheWrite. Note that reentrant calls are possible, i.e. the cache can call back the user (:arg:‘contp‘) in the same call.
TSConﬁgDataGet
Synopsis
#include  void* TSConfigDataGet(TSConﬁg conﬁgp)
Description
TSConﬁgGet
Synopsis
#include  TSConﬁg TSConfigGet(unsigned int id)
Description
TSConﬁgRelease
Synopsis
#include  void TSConfigRelease(unsigned int id, TSConﬁg conﬁgp)
4.9. API Reference 403 Apache-Trafﬁcserver-Server Documentation, latest
Description
TSConﬁgSet
Synopsis
#include  unsigned int TSConfigSet(unsigned int id, void * data, TSConﬁgDestroyFunc funcp)
Description
TSContCall
Synopsis
#include  int TSContCall(TSCont contp, TSEvent event, void * edata)
Description
TSContCreate
Synopsis
#include  TSCont TSContCreate(TSEventFunc funcp, TSMutex mutexp)
Description
TSContDataGet
Synopsis
#include  void* TSContDataGet(TSCont contp)
Description
TSContDataSet
Synopsis
#include  void TSContDataSet(TSCont contp, void * data)
404 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
TSContDestroy
Synopsis
#include  void TSContDestroy(TSCont contp)
Description
TSContMutexGet
Synopsis
#include  TSMutex TSContMutexGet(TSCont contp)
Description
TSContSchedule
Synopsis
#include  TSAction TSContSchedule(TSCont contp, ink_hrtime timeout, TSThreadPool tp)
Description
TSDebug
Trafﬁc Server Debugging APIs.
Synopsis
#include  void TSDebug(const char * tag, const char * format, ...) void TSError(const char * tag, const char * format, ...) int TSIsDebugTagSet(const char * tag) void TSDebugSpecific(int debug_ﬂag, const char * tag, const char * format, ...) void TSHttpTxnDebugSet(TSHttpTxn txnp, int on) void TSHttpSsnDebugSet(TSHttpSsn ssn, int on) int TSHttpTxnDebugGet(TSHttpTxn txnp)
4.9. API Reference 405 Apache-Trafﬁcserver-Server Documentation, latest
 int TSHttpSsnDebugGet(TSHttpSsn ssn) const char* TSHttpServerStateNameLookup(TSServerState state) const char* TSHttpHookNameLookup(TSHttpHookID hook) const char* TSHttpEventNameLookup(TSEvent event) void TSAssert(...) void TSReleaseAssert(...)
Description
TSError() is similar to printf except that instead of writing the output to the C standard output, it writes output to the Trafﬁc Server error log. TSDebug() is the same as TSError() except that it only logs the debug message if the given debug :arg:‘tag‘ is enabled. It writes output to the Trafﬁc Server debug log. TSIsDebugTagSet() returns non-zero if the given debug :arg:‘tag‘ is enabled. In debug mode, TSAssert Trafﬁc Server to prints the ﬁle name, line number and expression, and then aborts. In release mode, the expression is not removed but the effects of printing an error message and aborting are. TSReleaseAssert prints an error message and aborts in both release and debug mode. TSDebugSpecific() emits a debug line even if the debug :arg:‘tag‘ is turned off, as long as debug ﬂag is enabled. This can be used in conjunction with TSHttpTxnDebugSet(), TSHttpSsnDebugSet(), TSHttpTxnDebugGet() and TSHttpSsnDebugGet() to enable debugging on speciﬁc session and transac- tion objects. TSHttpServerStateNameLookup(), TSHttpHookNameLookup() and TSHttpEventNameLookup() converts the respective internal state to a string representation. This can be useful in debugging (TSDebug()), logging and other types notiﬁcations.
Examples
This example uses TSDebugSpecific() to log a message when a speciﬁc debugging ﬂag is enabled:
#include 
// Produce information about a hook receiving an event TSDebug(PLUGIN_NAME,"Entering hook= %s, event=%s", TSHttpHookNameLookup(hook), TSHttpEventNameLookup(event));
// Emit debug message if "tag" is enabled or the txn debug // flag is set. TSDebugSpecifc(TSHttpTxnDebugGet(txn),"tag", "Hello World from transaction%p", txn);
See Also
TSAPI(3ts), printf(3)
406 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSHostLookup
Synopsis
#include  TSAction TSHostLookup(TSCont contp, const char * hostname, size_t namelen)
Description
TSHostLookupResultAddrGet
Synopsis
#include  sockaddr const* TSHostLookupResultAddrGet(TSHostLookupResult lookup_result)
Description
TSHttpConnect
Synopsis
#include  TSVConn TSHttpConnect(sockaddr const * addr)
Description
Allows the plugin to initiate an HTTP connection. The TSVConn the plugin receives as the result of successful operates identically to one created through TSNetConnect. Aside from allowing the plugin to set the client ip and port for logging, the functional- ity of TSHttpConnect() is identical to connecting to localhost on the proxy port with TSNetConnect(). TSHttpConnect() is more efﬁcient than TSNetConnect() to localhost since it avoids the overhead of pass- ing the data through the operating system.
TSHttpConnectWithPluginId
Allows the plugin to initiate an http connection. This will tag the HTTP state machine with extra data that can be accessed by the logging interface. The connection is treated as an HTTP transaction as if it came from a client.
Synopsis
#include  TSVConn TSHttpConnectWithPluginId(sockaddr const * addr, char const * tag, int64_t id)
4.9. API Reference 407 Apache-Trafﬁcserver-Server Documentation, latest
Description
This call attempts to create an HTTP state machine and a virtual connection to that state machine. This is more efﬁcient than using TSNetConnect() because it avoids using the operating system stack via the loopback interface. :arg:‘addr‘ This is the network address of the target of the connection. This includes the port which should be stored in the sockaddr structure pointed at by :arg:‘addr‘. :arg:‘tag‘ This is a tag that is passed through to the HTTP state machine. It must be a persistent string that has a lifetime longer than the connection. It is accessible via the log ﬁeld pitag. This is intended as a class or type identiﬁer that is consistent across all connections for this plugin. In effect, the name of the plugin. This can be NULL. :arg:‘id‘ This is a numeric identiﬁer that is passed through to the HTTP state machine. It is accessible via the log ﬁeld piid. This is intended as a connection identiﬁer and should be distinct for every call to TSHttpConnectWithPluginId(). The easiest mechanism is to deﬁne a plugin global value and incre- ment it for each connection. The value 0 is reserved to mean “not set” and can be used as a default if this functionality is not needed. The virtual connection returned as the TSCVonn is API equivalent to a network virtual connection both to the plugin and to internal mechanisms. Data is read and written to the connection (and thence to the target system) by reading and writing on this virtual connection.
: This function only opens the connection. To drive the transaction an actual HTTP request must be sent and the HTTP response handled. The transaction is handled as a standard HTTP transaction and all of the standard conﬁguration options and plugins will operate on it.
The combination of :arg:‘tag‘ and :arg:‘id‘ is intended to enable correlation in log post processing. The :arg:‘tag‘ identiﬁes the connection as related to the plugin and the :arg:‘id‘ can be used in conjuction with plugin generated logs to correlate the log records.
Notes
The SPDY implementation uses this to correlate client sessions with SPDY streams. Each client connection is assigned a distinct numeric identiﬁer. This is passed as the :arg:‘id‘ to TSHttpConnectWithPluginId(). The :arg:‘tag‘ is selected to be the NPN string for the client session protocol, e.g. “spdy/3” or “spdy/3.1”. Log post processing can then count the number of connections for the various supported protocols and the number of SPDY virtual streams for each real client connection to Trafﬁc Server.
See Also
TSHttpConnect(3ts), TSNetConnect(3ts), TSAPI(3ts)
TSHttpHdrClone
Synopsis
#include  TSReturnCode TSHttpHdrClone(TSMBuffer dest_bufp, TSMBuffer src_bufp, TSMLoc src_hdr, TSMLoc * locp)
408 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
TSHttpHdrCopy
Copies the contents of the HTTP header located at :arg:‘src_loc‘ within :arg:‘src_bufp‘ to the HTTP header located at :arg:‘dest_loc‘ within :arg:‘dest_bufp‘.
Synopsis
#include  TSReturnCode TSHttpHdrCopy(TSMBuffer dest_bufp, TSMLoc dest_offset, TSMBuffer src_bufp, TSM- Loc src_offset)
Description
TSHttpHdrCopy() works correctly even if :arg:‘src_bufp‘ and :arg:‘dest_bufp‘ point to different Marshal Buffers. Make sure that you create the destination HTTP header before copying into it.
: TSHttpHdrCopy() appends the port number to the domain of the URL portion of the header. For example, a copy of http://www.example.com appears as http://www.example.com:80 in the destination buffer.
TSHttpHdrCreate
Synopsis
#include  TSMLoc TSHttpHdrCreate(TSMBuffer bufp)
Description
TSHttpHdrDestroy
Synopsis
#include  void TSHttpHdrDestroy(TSMBuffer bufp, TSMLoc offset)
Description
Destroys the HTTP header located at :arg:‘hdr_loc‘ within the marshal buffer :arg:‘bufp‘.
: Do not forget to release the handle :arg:‘hdr_loc‘ with a call to TSHandleMLocRelease().
4.9. API Reference 409 Apache-Trafﬁcserver-Server Documentation, latest
TSHttpHdrHostGet
Synopsis
#include  const char* TSHttpHdrHostGet(TSMBuffer bufp, TSMLoc offset, int * length)
Description
Get the host for the request. :arg:‘bufp‘ and :arg:‘offset‘ must reference an HTTP request header. A pointer to the host is returned and the length is stored in the int pointed at by :arg:‘length‘. Note the returned text may not be null terminated. The URL in the request is checked ﬁrst then the Host header ﬁeld.
: This is much faster than calling TSHttpEffectiveUrlStringGet() and extracting the host from the result.
TSHttpHdrLengthGet
Synopsis
#include  int TSHttpHdrLengthGet(TSMBuffer bufp, TSMLoc offset)
Description
TSHttpHdrMethodGet
Synopsis
#include  const char* TSHttpHdrMethodGet(TSMBuffer bufp, TSMLoc offset, int * length)
Description
TSHttpHdrMethodSet
Synopsis
#include  TSReturnCode TSHttpHdrMethodSet(TSMBuffer bufp, TSMLoc offset, const char * value, int length)
410 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
TSHttpHdrPrint
Synopsis
#include  void TSHttpHdrPrint(TSMBuffer bufp, TSMLoc offset, TSIOBuffer iobufp)
Description
TSHttpHdrReasonGet
Synopsis
#include  const char* TSHttpHdrReasonGet(TSMBuffer bufp, TSMLoc offset, int * length)
Description
TSHttpHdrReasonLookup
Synopsis
#include  const char* TSHttpHdrReasonLookup(TSHttpStatus status)
Description
TSHttpHdrReasonSet
Synopsis
#include  TSReturnCode TSHttpHdrReasonSet(TSMBuffer bufp, TSMLoc offset, const char * value, int length)
Description
TSHttpHdrStatusGet
Synopsis
#include  TSHttpStatus TSHttpHdrStatusGet(TSMBuffer bufp, TSMLoc offset)
4.9. API Reference 411 Apache-Trafﬁcserver-Server Documentation, latest
Description
TSHttpHdrStatusSet
Synopsis
#include  TSReturnCode TSHttpHdrStatusSet(TSMBuffer bufp, TSMLoc offset, TSHttpStatus status)
Description
TSHttpHdrTypeGet
Synopsis
#include  TSHttpType TSHttpHdrTypeGet(TSMBuffer bufp, TSMLoc offset)
Description
TSHttpHdrTypeSet
Synopsis
#include  TSReturnCode TSHttpHdrTypeSet(TSMBuffer bufp, TSMLoc offset, TSHttpType type)
Description
TSHttpHdrUrlGet
Synopsis
#include  TSReturnCode TSHttpHdrUrlGet(TSMBuffer bufp, TSMLoc offset, TSMLoc * locp)
Description
TSHttpHdrUrlSet
Synopsis
#include  TSReturnCode TSHttpHdrUrlSet(TSMBuffer bufp, TSMLoc offset, TSMLoc url)
412 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
TSHttpHdrVersionGet
Synopsis
#include  int TSHttpHdrVersionGet(TSMBuffer bufp, TSMLoc offset)
Description
TSHttpHdrVersionSet
Synopsis
#include  TSReturnCode TSHttpHdrVersionSet(TSMBuffer bufp, TSMLoc offset, int ver)
Description
TSHttpHookAdd
Intercept Trafﬁc Server events.
Synopsis
#include  void TSHttpHookAdd(TSHttpHookID id, TSCont contp) void TSHttpSsnHookAdd(TSHttpSsn ssnp, TSHttpHookID id, TSCont contp) void TSHttpTxnHookAdd(TSHttpTxn txnp, TSHttpHookID id, TSCont contp)
Description
Hooks are points in Apache Trafﬁc Server transaction HTTP processing where plugins can step in and do some work. Registering a plugin function for callback amounts to adding the function to a hook. You can register your plugin to be called back for every single transaction, or for speciﬁc transactions only. HTTP transaction hooks are set on a global basis using the function TSHttpHookAdd(). This means that the continuation speciﬁed as the parameter to TSHttpHookAdd() is called for every transaction. TSHttpHookAdd() is typically called from TSPluginInit() or TSRemapInit(). TSHttpSsnHookAdd() adds :arg:‘contp‘ to the end of the list of HTTP session hooks speciﬁed by :arg:‘id‘. This means that :arg:‘contp‘ is called back for every transaction within the session, at the point speciﬁed by the hook ID. Since :arg:‘contp‘ is added to a session, it is not possible to call TSHttpSsnHookAdd() from the plugin initialization routine; the plugin needs a handle to an HTTP session.
4.9. API Reference 413 Apache-Trafﬁcserver-Server Documentation, latest
TSHttpTxnHookAdd() adds :arg:‘contp‘ to the end of the list of HTTP transaction hooks speciﬁed by :arg:‘id‘. Since :arg:‘contp‘ is added to a transaction, it is not possible to call TSHttpTxnHookAdd() from the plugin initialization routine but only when the plugin has a handle to an HTTP transaction.
Return Values
None. Adding hooks is always successful.
Examples
The following example demonstrates how to add global, session and transaction hooks:
#include  static int handler(TSCont contp, TSEvent event, void *edata) { TSHttpSsn ssnp; TSHttpTxn txnp;
 switch (event){ case TS_EVENT_HTTP_SSN_START: ssnp= (TSHttpSsn) edata; // Add a session hook... TSHttpSsnHookAdd(ssnp, TS_HTTP_TXN_START_HOOK, contp); TSHttpSsnReenable(ssnp, TS_EVENT_HTTP_CONTINUE); return 0; case TS_EVENT_HTTP_TXN_START: txnp= (TSHttpTxn) edata; // Add a transaction hook... TSHttpTxnHookAdd(txnp, TS_HTTP_READ_REQUEST_HDR_HOOK, contp); TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE); return 0; default: break; }
 return 0; } void TSPluginInit (int argc, const char *argv[]) { TSCont contp; contp= TSContCreate(handler, NULL); TSHttpHookAdd(TS_HTTP_SSN_START_HOOK, contp); }
See Also
TSAPI(3ts), TSContCreate(3ts), TSLifecycleHookAdd(3ts)
414 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSHttpIsInternalRequest
Test whether a request is internally-generated.
Synopsis
#include  TSReturnCode TSHttpIsInternalRequest(TSHttpTxn txnp) TSReturnCode TSHttpIsInternalSession(TSHttpSsn ssnp)
Description
TSHttpIsInternalRequest() tests whether a HTTP transaction was originated within Trafﬁc Server.
Deprecated Deprecated in favor of TSHttpTxnIsInternal().
TSHttpIsInternalSession() tests whether a HTTP session was originated within Trafﬁc Server.
Deprecated Deprecated in favor of TSHttpSsnIsInternal().
Return Values
Both these APIs returns a TSReturnCode, indicating whether the object was internal (TS_SUCCESS) or not (TS_ERROR).
See Also
TSAPI(3ts), TSHttpSsnIsInternal(3ts), TSHttpTxnIsInternal(3ts)
TSHttpOverridableConﬁg
Synopsis
#include  TSOverridableConfigKey TSReturnCode TSHttpTxnConfigIntSet(TSHttpTxn txnp, TSOverridableConﬁgKey key, TSMgmtInt value) TSReturnCode TSHttpTxnConfigIntGet(TSHttpTxn txnp, TSOverridableConﬁgKey key, TSMgmtInt* value) TSReturnCode TSHttpTxnConfigFloatSet(TSHttpTxn txnp, TSOverridableConﬁgKey key, TSMgmt- Float value)
4.9. API Reference 415 Apache-Trafﬁcserver-Server Documentation, latest
TSReturnCode TSHttpTxnConfigFloatGet(TSHttpTxn txnp, TSOverridableConﬁgKey key, TSMgmt- Float* value) TSReturnCode TSHttpTxnConfigStringSet(TSHttpTxn txnp, TSOverridableConﬁgKey key, const char* value, int length) TSReturnCode TSHttpTxnConfigStringGet(TSHttpTxn txnp, TSOverridableConﬁgKey key, const char** value, int* length) TSReturnCode TSHttpTxnConfigFind(const char* name, int length, TSOverridableConﬁgKey* key, TSRecordDataType* type)
Description
Some of the values that are set in records.config can be changed for a speciﬁc transaction. It is important to note that these functions change the conﬁguration values stored for the transation, which is not quite the same as changing the actual operating values of the transaction. The critical effect is the value must be changed before it is used by the transaction - after that, changes will not have any effect. All of the ...Get functions store the internal value in the storage indicated by the :arg:‘value‘ argument. For strings :arg:‘length*‘ will receive the length of the string. The values are identiﬁed by the enumeration TSOverridableConfigKey. String values can be used indirectly by ﬁrst passing them to TSHttpTxnConfigFind() which, if the string matches an overridable value, return the key and data type.
Conﬁgurations
The following conﬁgurations (from records.config) are overridable.
:ts:cv:‘proxy.conﬁg.url_remap.pristine_host_hdr‘ :ts:cv:‘proxy.conﬁg.http.chunking_enabled‘ :ts:cv:‘proxy.conﬁg.http.negative_caching_enabled‘ :ts:cv:‘proxy.conﬁg.http.negative_caching_lifetime‘ :ts:cv:‘proxy.conﬁg.http.cache.when_to_revalidate‘ :ts:cv:‘proxy.conﬁg.http.keep_alive_enabled_in‘ :ts:cv:‘proxy.conﬁg.http.keep_alive_enabled_out‘ :ts:cv:‘proxy.conﬁg.http.keep_alive_post_out‘ :ts:cv:‘proxy.conﬁg.net.sock_recv_buffer_size_out‘ :ts:cv:‘proxy.conﬁg.net.sock_send_buffer_size_out‘ :ts:cv:‘proxy.conﬁg.net.sock_option_ﬂag_out‘ :ts:cv:‘proxy.conﬁg.http.forward.proxy_auth_to_parent‘ :ts:cv:‘proxy.conﬁg.http.anonymize_remove_from‘ :ts:cv:‘proxy.conﬁg.http.anonymize_remove_referer‘ :ts:cv:‘proxy.conﬁg.http.anonymize_remove_user_agent‘ :ts:cv:‘proxy.conﬁg.http.anonymize_remove_cookie‘ :ts:cv:‘proxy.conﬁg.http.anonymize_remove_client_ip‘ :ts:cv:‘proxy.conﬁg.http.anonymize_insert_client_ip‘ :ts:cv:‘proxy.conﬁg.http.response_server_enabled‘ :ts:cv:‘proxy.conﬁg.http.insert_squid_x_forwarded_for‘ :ts:cv:‘proxy.conﬁg.http.server_tcp_init_cwnd‘
416 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
:ts:cv:‘proxy.conﬁg.http.send_http11_requests‘ :ts:cv:‘proxy.conﬁg.http.cache.http‘ :ts:cv:‘proxy.conﬁg.http.cache.cluster_cache_local‘ :ts:cv:‘proxy.conﬁg.http.cache.ignore_client_no_cache‘ :ts:cv:‘proxy.conﬁg.http.cache.ignore_client_cc_max_age‘ :ts:cv:‘proxy.conﬁg.http.cache.ims_on_client_no_cache‘ :ts:cv:‘proxy.conﬁg.http.cache.ignore_server_no_cache‘ :ts:cv:‘proxy.conﬁg.http.cache.cache_responses_to_cookies‘ :ts:cv:‘proxy.conﬁg.http.cache.ignore_authentication‘ :ts:cv:‘proxy.conﬁg.http.cache.cache_urls_that_look_dynamic‘ :ts:cv:‘proxy.conﬁg.http.cache.required_headers‘ :ts:cv:‘proxy.conﬁg.http.insert_request_via_str‘ :ts:cv:‘proxy.conﬁg.http.insert_response_via_str‘ :ts:cv:‘proxy.conﬁg.http.cache.heuristic_min_lifetime‘ :ts:cv:‘proxy.conﬁg.http.cache.heuristic_max_lifetime‘ :ts:cv:‘proxy.conﬁg.http.cache.guaranteed_min_lifetime‘ :ts:cv:‘proxy.conﬁg.http.cache.guaranteed_max_lifetime‘ :ts:cv:‘proxy.conﬁg.http.cache.max_stale_age‘ :ts:cv:‘proxy.conﬁg.http.keep_alive_no_activity_timeout_in‘ :ts:cv:‘proxy.conﬁg.http.keep_alive_no_activity_timeout_out‘ :ts:cv:‘proxy.conﬁg.http.transaction_no_activity_timeout_in‘ :ts:cv:‘proxy.conﬁg.http.transaction_no_activity_timeout_out‘ :ts:cv:‘proxy.conﬁg.http.transaction_active_timeout_out‘ :ts:cv:‘proxy.conﬁg.websocket.no_activity_timeout‘ :ts:cv:‘proxy.conﬁg.websocket.active_timeout‘ :ts:cv:‘proxy.conﬁg.http.origin_max_connections‘ :ts:cv:‘proxy.conﬁg.http.connect_attempts_max_retries‘ :ts:cv:‘proxy.conﬁg.http.connect_attempts_max_retries_dead_server‘ :ts:cv:‘proxy.conﬁg.http.connect_attempts_rr_retries‘ :ts:cv:‘proxy.conﬁg.http.connect_attempts_timeout‘ :ts:cv:‘proxy.conﬁg.http.post_connect_attempts_timeout‘ :ts:cv:‘proxy.conﬁg.http.down_server.cache_time‘ :ts:cv:‘proxy.conﬁg.http.down_server.abort_threshold‘ :ts:cv:‘proxy.conﬁg.http.cache.fuzz.time‘ :ts:cv:‘proxy.conﬁg.http.cache.fuzz.min_time‘ :ts:cv:‘proxy.conﬁg.http.doc_in_cache_skip_dns‘ :ts:cv:‘proxy.conﬁg.http.background_ﬁll_active_timeout‘ :ts:cv:‘proxy.conﬁg.http.response_server_str‘ :ts:cv:‘proxy.conﬁg.http.cache.heuristic_lm_factor‘ :ts:cv:‘proxy.conﬁg.http.cache.fuzz.probability‘ :ts:cv:‘proxy.conﬁg.http.background_ﬁll_completed_threshold‘ :ts:cv:‘proxy.conﬁg.net.sock_packet_mark_out‘ :ts:cv:‘proxy.conﬁg.net.sock_packet_tos_out‘ :ts:cv:‘proxy.conﬁg.http.insert_age_in_response‘ :ts:cv:‘proxy.conﬁg.http.chunking.size‘ :ts:cv:‘proxy.conﬁg.http.ﬂow_control.enabled‘ :ts:cv:‘proxy.conﬁg.http.ﬂow_control.low_water‘
4.9. API Reference 417 Apache-Trafﬁcserver-Server Documentation, latest
:ts:cv:‘proxy.conﬁg.http.ﬂow_control.high_water‘ :ts:cv:‘proxy.conﬁg.http.cache.range.lookup‘ :ts:cv:‘proxy.conﬁg.http.normalize_ae_gzip‘ :ts:cv:‘proxy.conﬁg.http.default_buffer_size‘ :ts:cv:‘proxy.conﬁg.http.default_buffer_water_mark‘ :ts:cv:‘proxy.conﬁg.http.request_header_max_size‘ :ts:cv:‘proxy.conﬁg.http.response_header_max_size‘ :ts:cv:‘proxy.conﬁg.http.negative_revalidating_enabled‘ :ts:cv:‘proxy.conﬁg.http.negative_revalidating_lifetime‘ :ts:cv:‘proxy.conﬁg.http.accept_encoding_ﬁlter_enabled‘ :ts:cv:‘proxy.conﬁg.http.cache.range.write‘ :ts:cv:‘proxy.conﬁg.http.global_user_agent_header‘ :ts:cv:‘proxy.conﬁg.http.slow.log.threshold‘ :ts:cv:‘proxy.conﬁg.http.cache.generation‘ :ts:cv:‘proxy.conﬁg.body_factory.template_base‘ :ts:cv:‘proxy.conﬁg.http.cache.open_write_fail_action‘ :ts:cv:‘proxy.conﬁg.http.redirection_enabled‘ :ts:cv:‘proxy.conﬁg.http.number_of_redirections‘ :ts:cv:‘proxy.conﬁg.http.cache.max_open_write_retries‘ :ts:cv:‘proxy.conﬁg.http.redirect_use_orig_cache_key‘
Examples
Enable transaction buffer control with a high water mark of 262144 and a low water mark of 65536. int callback(TSCont contp, TSEvent event, void* data) { TSHttpTxn txnp= static_cast(data); TSHttpTxnConfigIntSet(txnp, TS_CONFIG_HTTP_FLOW_CONTROL_ENABLED,1); TSHttpTxnConfigIntSet(txnp, TS_CONFIG_HTTP_FLOW_CONTROL_HIGH_WATER_MARK, 262144); TSHttpTxnConfigIntSet(txnp, TS_CONFIG_HTTP_FLOW_CONTROL_LOWER_WATER_MARK, 65536); return 0; }
See Also
TSAPI(3ts)
TSHttpParserCreate
Parse HTTP headers from memory buffers.
Synopsis
#include  TSHttpParser TSHttpParserCreate(void)
418 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest void TSHttpParserClear(TSHttpParser parser) void TSHttpParserDestroy(TSHttpParser parser) TSParseResult TSHttpHdrParseReq(TSHttpParser parser, TSMBuffer bufp, TSMLoc offset, const char ** start, const char * end) TSParseResult TSHttpHdrParseResp(TSHttpParser parser, TSMBuffer bufp, TSMLoc offset, const char ** start, const char * end)
Description
TSHttpParserCreate() creates an HTTP parser object. The parser’s data structure contains information about the header being parsed. A single HTTP parser can be used multiple times, though not simultaneously. Before being used again, the parser must be cleared by calling TSHttpParserClear(). TSHttpHdrParseReq() parses an HTTP request header. The HTTP header :arg:‘offset‘ must already be created, and must reside inside the marshal buffer :arg:‘bufp‘. The :arg:‘start‘ argument points to the current position of the string buffer being parsed and the :arg:‘end‘ argument points to one byte after the end of the buffer to be parsed. On return, :arg:‘start‘ is modiﬁed to point past the last character parsed. It is possible to parse an HTTP request header a single byte at a time using repeated calls to TSHttpHdrParseReq(). As long as an error does not occur, the TSHttpHdrParseReq() func- tion will consume that single byte and ask for more. TSHttpHdrParseReq() should be called after TS_HTTP_READ_REQUEST_HDR_HOOK. TSHttpHdrParseResp() operates in the same manner as TSHttpHdrParseReq() except it parses an HTTP response header. It should be called after TS_HTTP_READ_RESPONSE_HDR_HOOK. TSHttpParserClear() clears the speciﬁed HTTP parser so it may be used again. TSHttpParserDestroy() destroys the TSHttpParser object pointed to by :arg:‘parser‘. The :arg:‘parser‘ pointer must not be NULL.
Return Salues
TSHttpHdrParseReq() and TSHttpHdrParseResp() both return a TSParseResult value. TS_PARSE_ERROR is returned on error, TS_PARSE_CONT is returned if parsing of the header stopped be- cause the end of the buffer was reached, and TS_PARSE_DONE or TS_PARSE_OK when a \r\n\r\n pattern is encountered, indicating the end of the header.
Known Bugs
The distinction between the TS_PARSE_DONE and TS_PARSE_OK results is not well-deﬁned. Plugins should expect both status codes and treat them equivalently.
See Also
TSAPI(3ts)
4.9. API Reference 419 Apache-Trafﬁcserver-Server Documentation, latest
TSHttpSsnClientFdGet
Synopsis
#include  TSReturnCode TSHttpSsnClientFdGet(TSHttpTxn txnp, int *fdp)
Description
Get the socket descriptor for the client socket in :arg:‘fdp‘ for the transaction :arg:‘ssn‘.
TSHttpSsnReenable
Synopsis
#include  void TSHttpSsnReenable(TSHttpSsn ssnp, TSEvent event)
Description
TSHttpTxnCacheLookupStatusGet
Synopsis
#include  TSReturnCode TSHttpTxnCacheLookupStatusGet(TSHttpTxn txnp, int * lookup_status)
Description
TSHttpTxnCacheLookupUrlGet
Synopsis
#include  TSReturnCode TSHttpTxnCacheLookupUrlGet(TSHttpTxn txnp, TSMBuffer bufp, TSMLoc offset)
Description
Get the current cache key URL, also referred to as the lookup URL. This must be stored in a properly allocated URL object, typically created with a TSUrlCreate().
420 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSHttpTxnCacheLookupUrlSet
Synopsis
#include  TSReturnCode TSHttpTxnCacheLookupUrlSet(TSHttpTxn txnp, TSMBuffer bufp, TSMLoc offset)
Description
Set the current cache key URL, also referred to as the lookup URL. This must be stored in a properly allocated URL object, typically created with a TSUrlCreate() or TSUrlClone(). This API can be called as early as TS_HTTP_READ_REQUEST_HDR_HOOK but no later than TS_HTTP_POST_REMAP_HOOK. This is the preferred and most efﬁcient way to modify the cache key, but an alternative is to use the old TSCacheUrlSet(), which takes a simple string as argument.
TSHttpTxnCachedReqGet
Synopsis
#include  TSReturnCode TSHttpTxnCachedReqGet(TSHttpTxn txnp, TSMBuffer * bufp, TSMLoc * offset)
Description
TSHttpTxnCachedRespGet
Synopsis
#include  TSReturnCode TSHttpTxnCachedRespGet(TSHttpTxn txnp, TSMBuffer * bufp, TSMLoc * offset)
Description
TSHttpTxnClientFdGet
Synopsis
#include  TSReturnCode TSHttpTxnClientFdGet(TSHttpTxn txnp, int *fdp)
Description
Get the socket descriptor for the client socket in :arg:‘fdp‘ for the transaction :arg:‘txnp‘.
4.9. API Reference 421 Apache-Trafﬁcserver-Server Documentation, latest
TSHttpTxnClientPacketDscpSet
Synopsis
#include  TSReturnCode TSHttpTxnClientPacketDscpSet(TSHttpTxn txnp, int dscp)
Description
Change packet :arg:‘dscp‘ for the client side connection.
: Changes take effect immediately.
See Also
Trafﬁc Shaping
TSHttpTxnClientPacketMarkSet
Synopsis
#include  TSReturnCode TSHttpTxnClientPacketMarkSet(TSHttpTxn txnp, int mark)
Description
Change packet ﬁrewall :arg:‘mark‘ for the client side connection.
: Changes take effect immediately.
See Also
TSHttpTxnClientPacketTosSet
Synopsis
#include  TSReturnCode TSHttpTxnClientPacketTosSet(TSHttpTxn txnp, int tos)
422 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
Change packet :arg:‘tos‘ for the client side connection.
: Changes take effect immediately
: TOS is deprecated and replaced by DSCP. This is still used to set DSCP, however the ﬁrst 2 bits of this value will be ignored as they now belong to the ECN ﬁeld.
See Also
TSHttpTxnClientReqGet
Synopsis
#include  TSReturnCode TSHttpTxnClientReqGet(TSHttpTxn txnp, TSMBuffer * bufp, TSMLoc * offset)
Description
TSHttpTxnClientRespGet
Synopsis
#include  TSReturnCode TSHttpTxnClientRespGet(TSHttpTxn txnp, TSMBuffer * bufp, TSMLoc * offset)
Description
TSHttpTxnErrorBodySet
Sets an error type body to a transaction.
Synopsis
#include  void TSHttpTxnErrorBodySet(TSHttpTxn txnp, char * buf, size_t buﬂength, char * mimetype)
Description
Note that both string arguments must be allocated with TSmalloc() or TSstrdup(). The :arg:‘mimetype‘ is optional, and if not provided it defaults to text/html. Sending an empty string would prevent setting a content type header (but that is not adviced).
4.9. API Reference 423 Apache-Trafﬁcserver-Server Documentation, latest
TSHttpTxnIncomingAddrGet
Get the incoming proxy address on which a client’s connection is accepted.
Synopsis
#include  sockaddr const* TSHttpTxnIncomingAddrGet(TSHttpTxn txnp)
Description
: The pointer is valid only for the current callback. Clients that need to keep the value across callbacks must maintain their own storage.
TSHttpTxnInfoIntGet
Synopsis
#include  TSReturnCode TSHttpTxnInfoIntGet(TSHttpTxn txnp, TSHttpTxnInfoKey key, TSMgmtInt *value)
Description
TSHttpTxnInfoIntGet() returns arbitrary integer-typed info about a transaction as deﬁned in TSHttpTxnInfoKey. The API will be part of a generic API umbrella that can support returning arbitrary info about a transaction using custom log tags. It works on multiple hooks depending on the requested info. For example, cache related info may be available only at or after TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK hook. The TSHttpTxnInfoKey currently supports the below integer-based info about a transaction TS_TXN_INFO_CACHE_HIT_RAM This info is available at or after TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK hook. A value of 1 indicates that the response is returned from RAM cache. A value of 0 indicates otherwise. TS_TXN_INFO_CACHE_COMPRESSED_IN_RAM This info is available at or after TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK hook. A value of 1 indicates that the response is returned from RAM cache and is compressed. A value of 0 indicates otherwise. TS_TXN_INFO_CACHE_HIT_RWW This info is available at or after TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK hook. A value of 1 indicates that the response is returned via Read-While-Writer functionality. A value of 0 indicates otherwise. TS_TXN_INFO_CACHE_OPEN_READ_TRIES
424 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
This info is available at or after TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK hook. The value indicates the number of cache open read reattempts made by the transaction on cache open read failure. TS_TXN_INFO_CACHE_OPEN_WRITE_TRIES This info is available at or after TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK hook. The value indicates the number of cache open write reattempts made by the transaction on cache open write failure. TS_TXN_INFO_CACHE_VOLUME This info is available at or after TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK hook. The value indicates the cache volume ID used for the cache object associated with the transaction.
Return values
The API returns TS_SUCCESS, if the requested info is supported, TS_ERROR otherwise.
TSHttpTxnIntercept
Allows a plugin take over the servicing of the request as though it was the origin server.
Synopsis
#include  void TSHttpTxnIntercept(TSCont contp, TSHttpTxn txnp)
Description
:arg:‘contp‘ will be sent TS_EVENT_NET_ACCEPT. The edata passed with TS_NET_EVENT_ACCEPT is an TSVConn just as it would be for a normal accept. The plugin must act as if it is an HTTP server and read the HTTP request and body off the TSVConn and send an HTTP response header and body. TSHttpTxnIntercept() must be called be called from only TS_HTTP_READ_REQUEST_HOOK. Using TSHttpTxnIntercept will bypass the Trafﬁc Server cache. If response sent by the plugin should be cached, use TSHttpTxnServerIntercept() instead. TSHttpTxnIntercept() primary use is allow plugins to serve data about their functioning directly. TSHttpTxnIntercept() must only be called once per transaction.
TSHttpTxnIsInternal
Test whether a request is internally-generated.
Synopsis
#include  TSReturnCode TSHttpTxnIsInternal(TSHttpTxn txnp) TSReturnCode TSHttpSsnIsInternal(TSHttpSsn ssnp)
4.9. API Reference 425 Apache-Trafﬁcserver-Server Documentation, latest
Description
TSHttpTxnIsInternal() tests whether a HTTP transaction was originated within Trafﬁc Server. TSHttpSsnIsInternal() tests whether a HTTP session was originated within Trafﬁc Server.
Return Values
Both these APIs returns a TSReturnCode, indicating whether the object was internal (TS_SUCCESS) or not (TS_ERROR).
Examples
The ESI plugin uses TSHttpTxnIsInternal() to ignore requests that is had generated while fetching portions of an ESI document:
See also
TSAPI(3ts)
TSHttpTxnMilestoneGet
Get a speciﬁed :arg:‘milestone‘ timer value for the current transaction.
Synopsis
#include  TSReturnCode TSHttpTxnMilestoneGet(TSHttpTxn txnp, TSMilestonesType milestone, TSHRTime * time)
Description
TSHttpTxnMilestoneGet() will fetch a speciﬁc :arg:‘milestone‘ timer value for the transaction :arg:‘txnp‘. These timers are calculated during the lifetime of a transaction and are measured in nanoseconds from the beginning of the transaction. :arg:‘time‘ is used a pointer to storage to update if the call is successful. TSMilestonesType
426 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Value Milestone The client connection is accepted. :const:‘TS_MILESTONE_UA_BEGIN‘ The request header from the client has been read and :const:‘TS_MILESTONE_UA_READ_HEADER_DONE‘ parsed. The response header write to the client starts. :const:‘TS_MILESTONE_UA_BEGIN_WRITE‘ Last I/O activity on the client socket, or connection abort. :const:‘TS_MILESTONE_UA_CLOSE‘ First time origin server connect attempted or shared shared :const:‘TS_MILESTONE_SERVER_FIRST_CONNECT‘ session attached. Most recent time origin server connect attempted or :const:‘TS_MILESTONE_SERVER_CONNECT‘ shared session attached. More recent time a connection attempt was resolved. :const:‘TS_MILESTONE_SERVER_CONNECT_END‘ First byte is written to the origin server connection. :const:‘TS_MILESTONE_SERVER_BEGIN_WRITE‘ First byte is read from connection to origin server. :const:‘TS_MILESTONE_SERVER_FIRST_READ‘ Origin server response has been read and parsed. :const:‘TS_MILESTONE_SERVER_READ_HEADER_DONE‘ Last I/O activity on origin server connection. :const:‘TS_MILESTONE_SERVER_CLOSE‘ Initiate read of the cache. :const:‘TS_MILESTONE_CACHE_OPEN_READ_BEGIN‘ Initial cache read has resolved. :const:‘TS_MILESTONE_CACHE_OPEN_READ_END‘ Start open for cache write. :const:‘TS_MILESTONE_CACHE_OPEN_WRITE_BEGIN‘ Cache has been opened for write. :const:‘TS_MILESTONE_CACHE_OPEN_WRITE_END‘ Initiate host resolution in HostDB :const:‘TS_MILESTONE_DNS_LOOKUP_BEGIN‘ Host resolution resolves. :const:‘TS_MILESTONE_DNS_LOOKUP_END‘ Transaction state machine is initialized. :const:‘TS_MILESTONE_SM_START‘ Transaction has ﬁnished, state machine ﬁnal logging has :const:‘TS_MILESTONE_SM_FINISH‘ started. Amount of time plugins were active plus start time. :const:‘TS_MILESTONE_PLUGIN_ACTIVE‘ Wall time while plugins were active plus start time. :const:‘TS_MILESTONE_PLUGIN_TOTAL‘ • The server connect times predate the transmission of the :literal:‘SYN packet. That is, before a connection to the origin server is completed. • A connection attempt is resolved when no more connection related activity remains to be done, and the connec- tion is either established or has failed. • :const:‘TS_MILESTONE_UA_CLOSE‘ and :const:‘TS_MILESTONE_SERVER_CLOSE‘ are updated continuously during the life of the transaction, every time there is I/O activity. The updating stops when the corresponding connection is closed, leaving the last I/O time as the ﬁnal value. • The cache OPEN milestones time only the initial setup, the open, not the full read or write. • :const:‘TS_MILESTONE_PLUGIN_ACTIVE‘ and :const:‘TS_MILESTONE_PLUGIN_TOTAL‘ are dif- ferent from the other milestones as they measure elapsed time, not event time. The value is the elapsed time plus :const:‘TS_MILESTONE_SM_START‘. This was decided to be more convenient because then these milestones can be handled / displayed in the same way as the other milestones, as offsets from :const:‘TS_MILESTONE_SM_START‘. :const:‘TS_MILESTONE_PLUGIN_ACTIVE‘ value is the amount of time the plugin was active, that is performing computation. :const:‘TS_MILESTONE_PLUGIN_TOTAL‘ is the wall time which in-
4.9. API Reference 427 Apache-Trafﬁcserver-Server Documentation, latest
 cludes any time the transaction was blocked while a plugin was active. For instance if a plugin waits on an external event, that waiting time will be in :const:‘TS_MILESTONE_PLUGIN_TOTAL‘ but not in :const:‘TS_MILESTONE_PLUGIN_ACTIVE‘.
Return Values
:const:‘TS_SUCCESS‘ if successful and :arg:‘time‘ was updated, otherwise :const:‘TS_ERROR‘.
See Also
TSAPI(3ts)
TSHttpTxnNextHopAddrGet
Synopsis
#include  sockaddr const * TSHttpTxnNextHopAddrGet(TSHttpTxn txnp)
Description
Get the next hop address for transaction :arg:‘txnp‘.
: The pointer is valid only for the current callback. Clients that need to keep the value across callbacks must maintain their own storage.
TSHttpTxnOutgoingAddrGet
Get the outgoing address used in origin connection.
Synopsis
#include  sockaddr const* TSHttpTxnOutgoingAddrGet(TSHttpTxn txnp)
Description
: The pointer is valid only for the current callback. Clients that need to keep the value across callbacks must maintain their own storage.
428 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSHttpTxnParentProxySet
Sets the parent proxy :arg:‘hostname‘ and :arg:‘port‘.
Synopsis
#include  void TSHttpTxnParentProxySet(TSHttpTxn txnp, const char * hostname, int port)
Description
The string :arg:‘hostname‘ is copied into the TSHttpTxn. You can modify or delete the string after calling TSHttpTxnParentProxySet().
TSHttpTxnReenable
Notiﬁes the HTTP transaction txnp that the plugin is ﬁnished processing the current hook.
Synopsis
#include  void TSHttpTxnReenable(TSHttpTxn txnp, TSEvent event)
Description
The plugin tells the transaction :arg:‘txnp‘ to either continue (TS_EVENT_HTTP_CONTINUE) or stop (TS_EVENT_HTTP_ERROR).
: You must always reenable the HTTP transaction after the processing of each transaction event. However, never reenable twice. Reenabling twice is a serious error.
TSHttpTxnServerAddrGet
Synopsis
#include  sockaddr const* TSHttpTxnServerAddrGet(TSHttpTxn txnp)
Description
Get the origin server address for transaction :arg:‘txnp‘.
4.9. API Reference 429 Apache-Trafﬁcserver-Server Documentation, latest
: The pointer is valid only for the current callback. Clients that need to keep the value across callbacks must maintain their own storage.
TSHttpTxnServerAddrSet
Synopsis
#include  TSReturnCode TSHttpTxnServerAddrGet(TSHttpTxn txnp, struct sockaddr const* addr)
Description
Set the origin server address for transaction :arg:‘txnp‘. This includes the port in :arg:‘addr‘. The address family is also set by the contents of :arg:‘addr‘. The address data is copied out of :arg:‘addr‘ so there is no dependency on the lifetime of that object. This hook must be called no later than TS_HTTP_OS_DNS_HOOK. If this is called then DNS resolution will not be done as the address of the server is already know. An error value is returned if :arg:‘addr‘ does not contain a valid IPv4 or IPv6 address with a valid (non-zero) port.
TSHttpTxnServerFdGet
Synopsis
#include  TSReturnCode TSHttpTxnServerFdGet(TSHttpTxn txnp, int *fdp)
Description
Get the socket descriptor for the server socket in :arg:‘fdp‘ for the transaction :arg:‘txnp‘.
TSHttpTxnServerIntercept
Intercept origin server requests.
Synopsis
#include  void TSHttpTxnServerIntercept(TSCont contp, TSHttpTxn txnp)
430 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
TSHttpTxnServerIntercept() allows a plugin take over the servicing of the request as though it was the origin server. In the event a request needs to be made to the server for transaction :arg:‘txnp‘, :arg:‘contp‘ will be sent a TS_EVENT_NET_ACCEPT event. The :arg:‘edata‘ passed with TS_NET_EVENT_ACCEPT is an TSVConn just as it would be for a normal accept. The plugin must act as if it is an HTTP server and read the HTTP request and body from the TSVConn and send an HTTP response header and body. TSHttpTxnServerIntercept() must be not be called after the connection to the server has taken place. This means that the last hook that it can be called from is TS_HTTP_READ_CACHE_HDR_HOOK. If a connection to the server is not necessary, the continuation :arg:‘contp‘ will be sent a TS_EVENT_NET_ACCEPT_FAILED event when the transaction completes. The response from the plugin is cached subject to standard and conﬁgured HTTP caching rules. Should the plugin wish the response not be cached, the plugin must use appropriate HTTP response headers to prevent caching. The primary purpose of TSHttpTxnServerIntercept() is allow plugins to provide gateways to other protocols or to allow to plugin to its own transport for the next hop to the server. TSHttpTxnServerIntercept() overrides parent cache conﬁguration. TSHttpTxnServerIntercept() must only be called once per transaction. The continuation :arg:‘contp‘ must have been created with a valid TSMutex.
See Also
TSAPI(3ts)
TSHttpTxnServerPacketDscpSet
Change packet DSCP for the server side connection.
Synopsis
#include  TSReturnCode TSHttpTxnServerPacketDscpSet(TSHttpTxn txnp, int dscp)
Description
: The change takes effect immediately. If no OS connection has been made, then this sets the mark that will be used. If an OS connection is established
See Also
Trafﬁc Shaping
TSHttpTxnServerPacketMarkSet
Change packet ﬁrewall mark for the server side connection.
4.9. API Reference 431 Apache-Trafﬁcserver-Server Documentation, latest
Synopsis
#include  TSReturnCode TSHttpTxnServerPacketMarkSet(TSHttpTxn txnp, int mark)
Description
: The change takes effect immediately. If no OS connection has been made, then this sets the mark that will be used. If an OS connection is established
See Also
TSHttpTxnServerPacketTosSet
Change packet TOS for the server side connection.
Synopsis
#include  TSReturnCode TSHttpTxnServerPacketTosSet(TSHttpTxn txnp, int tos)
Description
: The change takes effect immediately. If no OS connection has been made, then this sets the mark that will be used. If an OS connection is established TOS is deprecated and replaced by DSCP. This is still used to set DSCP, however the ﬁrst 2 bits of this value will be ignored as they now belong to the ECN ﬁeld.
See Also
TSHttpTxnServerReqGet
Synopsis
#include  TSReturnCode TSHttpTxnServerReqGet(TSHttpTxn txnp, TSMBuffer * bufp, TSMLoc * offset)
432 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
TSHttpTxnServerRespGet
Synopsis
#include  TSReturnCode TSHttpTxnServerRespGet(TSHttpTxn txnp, TSMBuffer * bufp, TSMLoc * offset)
Description
TSHttpTxnSsnGet
Synopsis
#include  TSHttpSsn TSHttpTxnSsnGet(TSHttpTxn txnp)
Description
TSHttpTxnTransformRespGet
Synopsis
#include  TSReturnCode TSHttpTxnTransformRespGet(TSHttpTxn txnp, TSMBuffer * bufp, TSMLoc * offset)
Description
TSHttpTxnTransformedRespCache
Synopsis
#include  void TSHttpTxnTransformedRespCache(TSHttpTxn txnp, int on)
Description
TSHttpTxnUntransformedRespCache
Synopsis
#include  void TSHttpTxnUntransformedRespCache(TSHttpTxn txnp, int on)
4.9. API Reference 433 Apache-Trafﬁcserver-Server Documentation, latest
Description
TSIOBufferBlockReadStart
Synopsis
#include  const char * TSIOBufferBlockReadStart(TSIOBufferBlock blockp, TSIOBufferReader readerp, int64_t * avail)
Description
TSIOBufferCopy
Synopsis
#include  int64_t TSIOBufferCopy(TSIOBuffer bufp, TSIOBufferReader readerp, int64_t length, int64_t offset)
Description
TSIOBufferCreate
Trafﬁc Server IO buffer API.
Synopsis
#include  TSIOBuffer TSIOBufferCreate(void) TSIOBuffer TSIOBufferSizedCreate(TSIOBufferSizeIndex index) void TSIOBufferDestroy(TSIOBuffer bufp) int64_t TSIOBufferWrite(TSIOBuffer bufp, const void * buf, int64_t length) void TSIOBufferProduce(TSIOBuffer bufp, int64_t nbytes) int64_t TSIOBufferWaterMarkGet(TSIOBuffer bufp) void TSIOBufferWaterMarkSet(TSIOBuffer bufp, int64_t water_mark)
Description
The TSIOBuffer data structure is the building block of the TSVConn abstraction. An IO buffer is composed of a list of buffer blocks which are reference counted so that they can reside in multiple buffers at the same time. This makes it extremely efﬁcient to copy data from one IO buffer to another using TSIOBufferCopy() since Trafﬁc Server only needs to copy pointers and adjust reference counts appropriately and not actually copy any data. However, applications should still strive to ensure data blocks are a reasonable size.
434 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
The IO buffer abstraction provides for a single writer and multiple readers. In order for the readers to have no knowl- edge of each other, they manipulate IO buffers through the TSIOBufferReader data structure. Since only a single writer is allowed, there is no corresponding TSIOBufferWriter data structure. The writer simply modiﬁes the IO buffer directly. TSIOBufferCreate() creates an empty TSIOBuffer. TSIOBufferSizedCreate() creates an empty TSIOBuffer with an initial capacity of :arg:‘index‘ bytes. TSIOBufferDestroy() destroys the IO buffer :arg:‘bufp‘. Since multiple IO buffers can share data, this does not necessarily free all of the data associated with the IO buffer but simply decrements the appropriate reference counts. TSIOBufferWrite() appends :arg:‘length‘ bytes from the buffer :arg:‘buf‘ to the IO buffer :arg:‘bufp‘ and returns the number of bytes successfully written into the IO buffer. TSIOBufferProduce() makes :arg:‘nbytes‘ of data available for reading in the IO buffer :arg:‘bufp‘. A com- mon pattern for writing to an IO buffer is to copy data into a buffer block and then call INKIOBufferProduce to make the new data visible to any readers. The watermark of an TSIOBuffer is the minimum number of bytes of data that have to be in the buffer before calling back any continuation that has initiated a read operation on this buffer. As a writer feeds data into the TSIOBuffer, no readers are called back until the amount of data reaches the watermark. Setting a watermark can improve perfor- mance because it avoids frequent callbacks to read small amounts of data. TSIOBufferWaterMarkGet() gets the current watermark for the IO buffer :arg:‘bufp‘. TSIOBufferWaterMarkSet() gets the current watermark for the IO buffer :arg:‘bufp‘ to :arg:‘water_mark‘ bytes.
See Also
TSAPI(3ts), TSIOBufferReaderAlloc(3ts)
TSInstallDirGet
Return Trafﬁc Server installation directories.
Synopsis
#include  const char * TSInstallDirGet(void) const char * TSConfigDirGet(void) const char * TSPluginDirGet(void)
Description
TSInstallDirGet() returns the path to the root of the Trafﬁc Server installation. TSConfigDirGet() and TSPluginDirGet() return the complete, absolute path to the conﬁguration directory and the plugin installation directory, respectively.
Return Values
These functions all return a NUL-terminated string that must not be modiﬁed or freed.
4.9. API Reference 435 Apache-Trafﬁcserver-Server Documentation, latest
Examples
To load a ﬁle that is located in the Trafﬁc Server conﬁguration directory:
#include  #include 
 char * path; asprintf(&path," %s/example.conf", TSConfigDirGet());
See Slso
TSAPI(3ts)
TSLifecycleHookAdd
Synopsis
#include  void TSLifecycleHookAdd(TSLifecycleHookID id, TSCont contp)
Description
TSLifecycleHookAdd() adds :arg:‘contp‘ to the list of lifecycle hooks speciﬁed by :arg:‘id‘. Lifecycle hooks are based on the Trafﬁc Server process, not on any speciﬁc transaction or session. These will typically be called only once during the execution of the Trafﬁc Server process and therefore should be added in TSPluginInit() (which could itself be considered a lifecyle hook). Unlike other hooks, lifecycle hooks may not have a well deﬁned ordering and use of them should not assume that one of the hooks is always called before another unless speciﬁcally mentioned.
TS_LIFECYCLE_PORTS_INITIALIZED_HOOK Called after the :ts:cv:‘proxy server port ‘ data structures have been initial- ized but before connections are accepted on those ports. The sockets corresponding to the ports may or may not be open depending on how the traffic_server process was invoked. Other API functions that depend on server ports should be called from this hook and not TSPluginInit(). Invoked with the event TS_EVENT_LIFECYCLE_PORTS_INITIALIZED and NULL data. TS_LIFECYCLE_PORTS_READY_HOOK Called after enabling connections on the proxy server ports. Because Trafﬁc Server is threaded this may or may not be called before any connections are accepted. The hook code may assume that any connection to Trafﬁc Server started after this hook is called will be accepted by Trafﬁc Server, making this a convenient place to signal external processes of that. Invoked with the event TS_EVENT_LIFECYCLE_PORTS_READY and NULL data. TS_LIFECYCLE_CACHE_READY_HOOK Called after Trafﬁc Server cache initialization has ﬁnished. Invoked with the event TS_EVENT_LIFECYCLE_CACHE_READY and NULL data.
436 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Ordering
TS_LIFECYCLE_PORTS_INITIALIZED_HOOK will always be called before TS_LIFECYCLE_PORTS_READY_HOOK.
Examples
The following example demonstrates how to correctly use TSNetAcceptNamedProtocol(), which requires the proxy ports to be initialized and therefore does not work if called from TSPluginInit() directly.
#include 
#define SSL_PROTOCOL_NAME "whatever"
 static int ssl_proto_handler(TSCont contp, TSEvent event, void* data) { /// Do named protocol handling. }
 static int local_ssl_init(TSCont contp, TSEvent event, void * edata) { if (TS_EVENT_LIFECYCLE_PORTS_INITIALIZED== event) { // just to be safe. TSNetAcceptNamedProtocol( TSContCreate(ssl_proto_handler, TSMutexCreate()), SSL_PROTOCOL_NAME ); } return 0; }
 void TSPluginInit( int argc, const char * argv[]) { TSLifecycleHookAdd(TS_LIFECYCLE_PORTS_INITIALIZED_HOOK, TSContCreate(local_ssl_
˓→init, NULL)); }
History
Lifecycle hooks were introduced to solve process initialization ordering issues (TS-1487). Different API calls required different modules of Trafﬁc Server to be initialized for the call to work, but others did not work that late in initialization, which was problematic because all of them could effectively only be called from TSPluginInit() . The solution was to move TSPluginInit() as early as possible in the process initialization and provide hooks for API calls that needed to be invoked later which served essentially as additional pluging initialization points.
See Also
TSAPI(3ts), TSContCreate(3ts)
4.9. API Reference 437 Apache-Trafﬁcserver-Server Documentation, latest
TSMBufferCreate
Trafﬁc Server marshall buffer API.
Synopsis
#include  TSMBuffer TSMBufferCreate(void) TSReturnCode TSMBufferDestroy(TSMBuffer bufp) TSReturnCode TSHandleMLocRelease(TSMBuffer bufp, TSMLoc parent, TSMLoc mloc)
Description
The marshal buffer or TSMBuffer is a heap data structure that stores parsed URLs, MIME headers and HTTP head- ers. You can allocate new objects out of marshal buffers, and change the values within the marshal buffer. Whenever you manipulate an object, you require the handle to the object (TSMLoc) and the marshal buffer containing the object (TSMBuffer). Any marshal buffer fetched by TSHttpTxn*Get() will be used by other parts of the system. Be careful not to destroy these shared, transaction marshal buffers. TSMBufferCreate() creates a new marshal buffer and initializes the reference count. TSMBufferDestroy() Ignores the reference count and destroys the marshal buffer bufp. The internal data buffer associated with the marshal buffer is also destroyed if the marshal buffer allocated it. TSHandleMLocRelease() Releases the TSMLoc mloc created from the TSMLoc parent. If a TSMLoc is obtained from a transaction, it does not have a parent TSMLoc. Use the the constant TS_NULL_MLOC as its parent.
Return values
TSMBufferDestroy() and TSHandleMLocRelease() return TS_SUCCESS on success, or TS_ERROR on failure. TSMBufferCreate() returns the new TSMBuffer.
Examples
#include  static void copyResponseMimeHdr (TSCont pCont, TSHttpTxn pTxn) { TSMBuffer respHdrBuf, tmpBuf; TSMLoc respHttpHdrLoc, tmpMimeHdrLoc;
 if (!TSHttpTxnClientRespGet(pTxn, &respHdrBuf, &respHttpHdrLoc)) { TSError("couldn't retrieve client response header0); TSHandleMLocRelease(respHdrBuf, TS_NULL_MLOC, respHttpHdrLoc); goto done; }
 tmpBuf = TSMBufferCreate(); tmpMimeHdrLoc = TSMimeHdrCreate(tmpBuf);
438 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSMimeHdrCopy(tmpBuf, tmpMimeHdrLoc, respHdrBuf, respHttpHdrLoc); TSHandleMLocRelease(tmpBuf, TS_NULL_MLOC, tmpMimeHdrLoc); TSHandleMLocRelease(respHdrBuf, TS_NULL_MLOC, respHttpHdrLoc); TSMBufferDestroy(tmpBuf);
 done: TSHttpTxnReenable(pTxn, TS_EVENT_HTTP_CONTINUE); }
See also
TSAPI(3ts)
TSMgmtCounterGet
Synopsis
#include  TSReturnCode TSMgmtCounterGet(const char * var_name, TSMgmtCounter * result)
Description
TSMgmtFloatGet
Synopsis
#include  TSReturnCode TSMgmtFloatGet(const char * var_name, TSMgmtFloat * result)
Description
TSMgmtIntGet
Synopsis
#include  TSReturnCode TSMgmtIntGet(const char * var_name, TSMgmtInt * result)
Description
TSMgmtStringGet
Synopsis
#include  TSReturnCode TSMgmtStringGet(const char * var_name, TSMgmtString * result)
4.9. API Reference 439 Apache-Trafﬁcserver-Server Documentation, latest
Description
TSMgmtUpdateRegister
Synopsis
#include  void TSMgmtUpdateRegister(TSCont contp, const char * plugin_name)
Description
TSMimeHdrClone
Synopsis
#include  TSReturnCode TSMimeHdrClone(TSMBuffer dest_bufp, TSMBuffer src_bufp, TSMLoc src_hdr, TSMLoc * locp)
Description
Copies a speciﬁed MIME header to a speciﬁed marshal buffer, and returns the location of the copied MIME header within the destination marshal buffer. Unlike TSMimeHdrCopy(), you do not have to create the destination MIME header before cloning. Release the returned TSMLoc handle with a call to TSHandleMLocRelease().
TSMimeHdrCopy
Synopsis
#include  TSReturnCode TSMimeHdrCopy(TSMBuffer dest_bufp, TSMLoc dest_offset, TSMBuffer src_bufp, TSM- Loc src_offset)
Description
Copies the contents of the MIME header located at :arg:‘src_loc‘ within :arg:‘src_bufp‘ to the MIME header located at :arg:‘dest_loc‘ within :arg:‘dest_bufp‘. TSMimeHdrCopy() works correctly even if :arg:‘src_bufp‘ and :arg:‘dest_bufp‘ point to different marshal buffers.
: You must create the destination MIME header before copying into it, using TSMimeHdrCreate().
440 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSMimeHdrCreate
Synopsis
#include  TSReturnCode TSMimeHdrCreate(TSMBuffer bufp, TSMLoc * locp)
Description
Creates a new MIME header within :arg:‘bufp‘. Release with a call to TSHandleMLocRelease().
TSMimeHdrDestroy
Synopsis
#include  TSReturnCode TSMimeHdrDestroy(TSMBuffer bufp, TSMLoc offset)
Description
Destroys the MIME header located at :arg:‘hdr_loc‘ within :arg:‘bufp‘.
TSMimeHdrFieldAppend
Synopsis
#include  TSReturnCode TSMimeHdrFieldAppend(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld)
Description
Returns the TSMLoc location of a speciﬁed MIME ﬁeld from within the MIME header located at :arg:‘hdr‘. The retrieved_str parameter speciﬁes which ﬁeld to retrieve. For each MIME ﬁeld in the MIME header, a pointer comparison is done between the ﬁeld name and retrieved_str. This is a much quicker retrieval function than TSMimeHdrFieldFind() since it obviates the need for a string comparison. However, retrieved_str must be one of the predeﬁned ﬁeld names of the form TS_MIME_FIELD_XXX for the call to succeed. Release the returned TSMLoc handle with a call to TSHandleMLocRelease().
TSMimeHdrFieldClone
Synopsis
#include 
4.9. API Reference 441 Apache-Trafﬁcserver-Server Documentation, latest
TSReturnCode TSMimeHdrFieldClone(TSMBuffer dest_bufp, TSMLoc dest_hdr, TSMBuffer src_bufp, TSMLoc src_hdr, TSMLoc src_ﬁeld, TSMLoc * locp)
Description
TSMimeHdrFieldCopy
Synopsis
#include  TSReturnCode TSMimeHdrFieldCopy(TSMBuffer dest_bufp, TSMLoc dest_hdr, TSMLoc dest_ﬁeld, TSM- Buffer src_bufp, TSMLoc src_hdr, TSMLoc src_ﬁeld)
Description
TSMimeHdrFieldCopyValues
Synopsis
#include  TSReturnCode TSMimeHdrFieldCopyValues(TSMBuffer dest_bufp, TSMLoc dest_hdr, TSM- Loc dest_ﬁeld, TSMBuffer src_bufp, TSMLoc src_hdr, TSMLoc src_ﬁeld)
Description
TSMimeHdrFieldCreate
Synopsis
#include  TSReturnCode TSMimeHdrFieldCreate(TSMBuffer bufp, TSMLoc hdr, TSMLoc * locp)
Description
TSMimeHdrFieldDestroy
Synopsis
#include  TSReturnCode TSMimeHdrFieldDestroy(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld)
442 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
Destroys the MIME :arg:‘ﬁeld‘ located at :arg:‘ﬁeld‘ within :arg:‘bufp‘. You must release the TSMLoc ﬁeld with a call to TSHandleMLocRelease().
TSMimeHdrFieldFind
Synopsis
#include  TSMLoc TSMimeHdrFieldFind(TSMBuffer bufp, TSMLoc hdr, const char * name, int length)
Description
Retrieves the TSMLoc location of a speciﬁed MIME ﬁeld from within the MIME header located at :arg:‘hdr‘. The :arg:‘name‘ and :arg:‘length‘ parameters specify which ﬁeld to retrieve. For each MIME ﬁeld in the MIME header, a case insensitive string comparison is done between the ﬁeld name and :arg:‘name‘. If TSMimeHdrFieldFind() cannot ﬁnd the requested ﬁeld, it returns TS_NULL_MLOC. Release the returned TSMLoc handle with a call to TSHandleMLocRelease().
TSMimeHdrFieldGet
Synopsis
#include  TSMLoc TSMimeHdrFieldGet(TSMBuffer bufp, TSMLoc hdr, int idx)
Description
Retrieves the location of a speciﬁed MIME ﬁeld within the MIME header located at :arg:‘hdr_loc‘ within :arg:‘bufp‘. The :arg:‘idx‘ parameter speciﬁes which ﬁeld to retrieve. The ﬁelds are numbered from 0 to TSMimeHdrFieldsCount(bufp, hdr_loc) - 1. If :arg:‘idx‘ does not lie within that range then TSMimeHdrFieldGet returns 0. Release the returned handle with a call to TSHandleMLocRelease.
TSMimeHdrFieldLengthGet
Synopsis
#include  int TSMimeHdrFieldLengthGet(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld)
4.9. API Reference 443 Apache-Trafﬁcserver-Server Documentation, latest
Description
TSMimeHdrFieldNameGet
Synopsis
#include  const char * TSMimeHdrFieldNameGet(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld, int * length)
Description
TSMimeHdrFieldNameSet
Synopsis
#include  TSReturnCode TSMimeHdrFieldNameSet(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld, const char * name, int length)
Description
TSMimeHdrFieldNext
Synopsis
#include  TSMLoc TSMimeHdrFieldNext(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld)
Description
TSMimeHdrFieldNextDup
Synopsis
#include  TSMLoc TSMimeHdrFieldNextDup(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld)
Description
TSMimeHdrFieldRemove
Synopsis
#include  TSReturnCode TSMimeHdrFieldRemove(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld)
444 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
Removes the MIME ﬁeld located at :arg:‘ﬁeld‘ within :arg:‘bufp‘ from the header located at :arg:‘hdr‘ within :arg:‘bufp‘. If the speciﬁed ﬁeld cannot be found in the list of ﬁelds associated with the header then nothing is done.
: Removing the ﬁeld does not destroy the ﬁeld. It only detaches the ﬁeld, hiding it from the printed output. The ﬁeld can be reattached with a call to TSMimeHdrFieldAppend(). If you do not use the detached ﬁeld you should destroy it with a call to TSMimeHdrFieldDestroy() and release the handle ﬁeld with a call to TSHandleMLocRelease().
TSMimeHdrFieldValueAppend
Synopsis
#include  TSReturnCode TSMimeHdrFieldValueAppend(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld, int idx, const char * value, int length)
Description
TSMimeHdrFieldValueDateInsert
Synopsis
#include  TSReturnCode TSMimeHdrFieldValueDateInsert(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld, time_t value)
Description
TSMimeHdrFieldValueDateSet
Synopsis
#include  TSReturnCode TSMimeHdrFieldValueDateSet(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld, time_t value)
Description
TSMimeHdrFieldValueIntSet
4.9. API Reference 445 Apache-Trafﬁcserver-Server Documentation, latest
Synopsis
#include  TSReturnCode TSMimeHdrFieldValueIntSet(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld, int idx, int value)
Description
TSMimeHdrFieldValueStringGet
Get HTTP MIME header values.
Synopsis
#include  const char * TSMimeHdrFieldValueStringGet(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld, int idx, int * value_len_ptr) int TSMimeHdrFieldValueIntGet(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld, int idx) int64_t TSMimeHdrFieldValueInt64Get(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld, int idx) unsigned int TSMimeHdrFieldValueUintGet(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld, int idx) time_t TSMimeHdrFieldValueDateGet(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld)
Description
MIME headers and ﬁelds can be components of request headers, response headers, or standalone headers cre- ated within a Trafﬁc Server plugin. The functions here are all used to access header values of speciﬁc types, but it is up to the caller to know if a header has appropriate semantics for the API used. For all but TSMimeHdrFieldValueStringGet(), an appropriate data conversion algorithm is applied to the header ﬁeld string. All the APIs take a TSMBuffer marshal buffer argument :arg:‘bufp‘, and a TSMLoc argument :arg:‘hdr‘ indicating the location of the HTTP headers. The required :arg:‘ﬁeld‘ argument is the locator of a speciﬁc header value, as returned by an accessor function such as TSMimeHdrFieldFind(). Within the header ﬁeld, comma-separated values can be retrieved with an index :arg:‘idx‘ ranging from 0 to the maximum number of ﬁelds for this value; this maximum is retrieved using TSMimeHdrFieldValuesCount(). An :arg:‘idx‘ value of -1 has the semantics of retrieving the entire header value, regardless of how many comma- separated values there are. If a header is not comma-separated, an :arg:‘idx‘ of 0 or -1 are the same, but the latter is preferred. TSMimeHdrFieldValueStringGet() returns a pointer to the header value, and populated :arg:‘value_len_ptr‘ with the length of the value in bytes. The returned header value is not NUL-terminated.
Return Values
All functions returns the header value with a type matching the respective function name. Using TSMimeHdrFieldValueDateGet() on a header which does not have date-time semantics always returns 0.
446 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Examples
This examples show how to retrieve and copy a speciﬁc header.
#include  #include  int get_content_type(TSHttpTxn txnp, char* buf, size_t buf_size) { TSMBuffer bufp; TSMLoc hdrs; TSMLoc ctype_field; int len=-1;
 if (TS_SUCCESS == TSHttpTxnServerRespGet(txnp,&bufp,&hdrs)) { ctype_field= TSMimeHdrFieldFind(bufp, hdrs, TS_MIME_FIELD_CONTENT_TYPE, TS_MIME_
˓→LEN_CONTENT_TYPE);
 if (TS_NULL_MLOC != ctype_field) { const char* str= TSMimeHdrFieldValueStringGet(bufp, hdrs, ctype_field,-1,& ˓→len);
 if (len> buf_size) len= buf_size; memcpy(buf, str, len); TSHandleMLocRelease(bufp, hdrs, ctype_field); } TSHandleMLocRelease(bufp, TS_NULL_MLOC, hdrs); }
 return len; }
See Slso
TSAPI(3ts), TSMBufferCreate(3ts), TSMimeHdrFieldValuesCount(3ts)
TSMimeHdrFieldValueStringInsert
Synopsis
#include  TSReturnCode TSMimeHdrFieldValueStringInsert(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld, int idx, const char * value, int length)
Description
TSMimeHdrFieldValueStringSet
4.9. API Reference 447 Apache-Trafﬁcserver-Server Documentation, latest
Synopsis
#include  TSReturnCode TSMimeHdrFieldValueStringSet(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld, int idx, const char * value, int length)
Description
TSMimeHdrFieldValueUintInsert
Synopsis
#include  TSReturnCode TSMimeHdrFieldValueUintInsert(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld, int idx, unsigned int value)
Description
TSMimeHdrFieldValueUintSet
Synopsis
#include  TSReturnCode TSMimeHdrFieldValueUintSet(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld, int idx, unsigned int value)
Description
TSMimeHdrFieldValuesClear
Synopsis
#include  TSReturnCode TSMimeHdrFieldValuesClear(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld)
Description
TSMimeHdrFieldValuesCount
Synopsis
#include  int TSMimeHdrFieldValuesCount(TSMBuffer bufp, TSMLoc hdr, TSMLoc ﬁeld)
448 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
TSMimeHdrFieldsClear
Synopsis
#include  TSReturnCode TSMimeHdrFieldsClear(TSMBuffer bufp, TSMLoc offset)
Description
Removes and destroys all the MIME ﬁelds within the MIME header located at :arg:‘hdr_loc‘ within the marshal buffer :arg:‘bufp‘.
TSMimeHdrFieldsCount
Synopsis
#include  int TSMimeHdrFieldsCount(TSMBuffer bufp, TSMLoc offset)
Description
Returns a count of the number of MIME ﬁelds within the MIME header located at :arg:‘hdr_loc‘ within the marshal buffer :arg:‘bufp‘.
TSMimeHdrLengthGet
Synopsis
#include  int TSMimeHdrLengthGet(TSMBuffer bufp, TSMLoc offset)
Description
Calculates the length of the MIME header located at :arg:‘hdr_loc‘ in buffer :arg:‘bufp‘ if it were returned as a string, in its unparsed form.
TSMimeHdrParse
Synopsis
#include  TSParseResult TSMimeHdrParse(TSMimeParser parser, TSMBuffer bufp, TSMLoc offset, const char ** start, const char * end)
4.9. API Reference 449 Apache-Trafﬁcserver-Server Documentation, latest
Description
Parses a MIME header. The MIME header must have already been allocated and both :arg:‘bufp‘ and :arg:‘hdr_loc‘ must point within that header. It is possible to parse a MIME header a single byte at a time using repeated calls to TSMimeHdrParse(). As long as an error does not occur, TSMimeHdrParse() consumes each single byte and asks for more.
TSMimeHdrPrint
Synopsis
#include  void TSMimeHdrPrint(TSMBuffer bufp, TSMLoc offset, TSIOBuffer iobufp)
Description
Formats the MIME header located at :arg:‘hdr_loc‘ within :arg:‘bufp‘ into the TSIOBuffer :arg:‘iobufp‘.
TSMimeParserClear
Synopsis
#include  void TSMimeParserClear(TSMimeParser parser)
Description
Clears the speciﬁed MIME :arg:‘parser‘ so that it can be used again.
TSMimeParserCreate
Synopsis
#include  TSMimeParser TSMimeParserCreate(void)
Description
Creates a MIME parser. The parser’s data structure contains information about the header being parsed. A single MIME parser can be used multiple times, though not simultaneously. Before being used again, the parser must be cleared by calling TSMimeParserClear().
450 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSMimeParserDestroy
Synopsis
#include  void TSMimeParserDestroy(TSMimeParser parser)
Description
Destroys the speciﬁed MIME :arg:‘parser‘ and frees the associated memory.
TSMutexCreate
Synopsis
#include  TSMutex TSMutexCreate(void)
Description
TSMutexDestroy
Synopsis
#include  void TSMutexDestroy(TSMutex mutexp)
Description
Destroys the indicated :arg:‘mutex‘ previously created via TSMutexCreate().
TSMutexLock
Synopsis
#include  void TSMutexLock(TSMutex mutexp)
Description
TSMutexLockTry
Synopsis
#include 
4.9. API Reference 451 Apache-Trafﬁcserver-Server Documentation, latest
TSReturnCode TSMutexLockTry(TSMutex mutexp)
Description
TSMutexUnlock
Synopsis
#include  void TSMutexUnlock(TSMutex mutexp)
Description
TSNetAccept
Synopsis
#include  TSAction TSNetAccept(TSCont contp, int port, int domain, int accept_threads)
Description
TSNetAcceptNamedProtocol
Listen on all SSL ports for connections for the speciﬁed protocol name.
Synopsis
#include  TSReturnCode TSNetAcceptNamedProtocol(TSCont contp, const char * protocol)
Description
TSNetAcceptNamedProtocol registers the speciﬁed :arg:‘protocol‘ for all statically conﬁgured TLS ports. When a client using the TLS Next Protocol Negotiation extension negotiates the requested protocol, TrafficServer will route the request to the given handler :arg:‘contp‘.
: Be aware that the protocol is not registered on ports opened by other plugins.
The event and data provided to the handler are the same as for TSNetAccept(). If a connection is successfully accepted, the event code will be TS_EVENT_NET_ACCEPT and the event data will be a valid TSVConn bound to the accepted connection.
: Neither :arg:‘contp‘ nor :arg:‘protocol‘ are copied. They must remain valid for the lifetime of the plugin.
452 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSNetAcceptNamedProtocol fails if the requested protocol cannot be registered on all of the conﬁgured TLS ports. If it fails, the protocol will not be registered on any ports. There is no partial failure.
TSNetConnect
Synopsis
#include  TSAction TSNetConnect(TSCont contp, sockaddr const * addr)
Description
TSPluginInit
Trafﬁc Server plugin loading and registration.
Synopsis
#include  void TSPluginInit(int argc, const char* argv[]) TSReturnCode TSPluginRegister(TSPluginRegistrationInfo* plugin_info)
Description
TSPluginInit() must be deﬁned by all plugins. Trafﬁc Server calls this initialization routine when it loads the plugin and sets :arg:‘argc‘ and :arg:‘argv‘ appropriately based on the values in plugin.config. :arg:‘argc‘ is a count of the number of arguments in the argument vector, :arg:‘argv‘. The count is at least one because the ﬁrst argument in the argument vector is the plugins name, which must exist in order for the plugin to be loaded. TSPluginRegister() registers the appropriate SDK version speciﬁc in :arg:‘sdk_version‘ for your plugin. Use this function to make sure that the version of Trafﬁc Server on which your plugin is running supports the plugin.
Return Values
TSPluginRegister() returns :const:‘TS_ERROR‘ if the plugin registration failed.
Examples
#include 
 void TSPluginInit (int argc, const char *argv[]) { TSPluginRegistrationInfo info; info.plugin_name="hello-world";
4.9. API Reference 453 Apache-Trafﬁcserver-Server Documentation, latest
 info.vendor_name="MyCompany"; info.support_email="[email protected]";
 if (TSPluginRegister(&info) != TS_SUCCESS) { TSError("Plugin registration failed. 0); } }
See Also
TSAPI(3ts), TSInstallDirGet(3ts)
TSRemapInit
Trafﬁc Server remap plugin entry points.
Synopsis
#include  #include  TSReturnCode TSRemapInit(TSRemapInterface * api_info, char * errbuf, int errbuf_size) void TSRemapDone(void) TSRemapStatus TSRemapDoRemap(void * ih, TSHttpTxn rh, TSRemapRequestInfo * rri) TSReturnCode TSRemapNewInstance(int argc, char * argv[], void ** ih, char * errbuf, int errbuf_size) void TSRemapDeleteInstance(void *) void TSRemapOSResponse(void * ih, TSHttpTxn rh, int os_response_type)
Description
The Trafﬁc Server remap interface provides a simpliﬁed mechanism for plugins to manipulate HTTP transactions. A remap plugin is not global; it is conﬁgured on a per-remap rule basis, which enables you to customize how URLs are redirected based on individual rules in remap.config. Writing a remap plugin consists of implementing one or more of the remap entry points and conﬁguring remap.config to route the transaction through your plugin. Multiple remap plugins can be speciﬁed for a single remap rule, resulting in a remap plugin chain where each plugin is given an opportunity to examine the HTTP transaction. TSRemapInit() is a required entry point. This function will be called once when Trafﬁc Server loads the plugin. If the optional TSRemapDone() entry point is available, Trafﬁc Server will call then when unloading the remap plugin. A remap plugin may be invoked for different remap rules. Trafﬁc Server will call the entry point each time a plugin is speciﬁed in a remap rule. When a remap plugin instance is no longer required, Trafﬁc Server will call TSRemapDeleteInstance(). TSRemapDoRemap() is called for each HTTP transaction. This is a mandatory entry point. In this function, the remap plugin may examine and modify the HTTP transaction.
454 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Return Values
TSRemapInit() and TSRemapNewInstance() should return TS_SUCCESS on success, and TS_ERROR oth- erwise. A return value of TS_ERROR is unrecoverable. TSRemapDoRemap() returns a status code that indicates whether the HTTP transaction has been modiﬁed and whether Trafﬁc Server should continue to evaluate the chain of remap plugins. If the transaction was modiﬁed, the plugin should return TSREMAP_DID_REMAP or TSREMAP_DID_REMAP_STOP; otherwise it should return TSREMAP_NO_REMAP or TSREMAP_NO_REMAP_STOP. If Trafﬁc Server should not send the transaction to subse- quent plugins in the remap chain, return TSREMAP_NO_REMAP_STOP or TSREMAP_DID_REMAP_STOP. Return- ing TSREMAP_ERROR causes Trafﬁc Server to stop evaluating the remap chain and respond with an error.
See Also
TSAPI(3ts)
TSSslContextFindByName
Synopsis
#include  TSSslContext TSSslContextFindByName(const char * name)
Description
Look for a SSL context created from ssl_multicert.config. Use the server :arg:‘name‘ to search.
TSSslContextFindByAddr
Synopsis
#include  TSSslContext TSSslContextFindByAddr(struct sockaddr const*)
Description
Look for a SSL context created from ssl_multicert.config. Use the server address to search.
Type
TSSslContext Corresponds to the SSL_CTX * value in openssl.
TSTextLogObjectCreate
Trafﬁc Server text logging API.
4.9. API Reference 455 Apache-Trafﬁcserver-Server Documentation, latest
Synopsis
#include  TSReturnCode TSTextLogObjectCreate(const char * ﬁlename, int mode, TSTextLogObject * new_log_obj) TSReturnCode TSTextLogObjectWrite(TSTextLogObject the_object, const char * format, ...) void TSTextLogObjectFlush(TSTextLogObject the_object) TSReturnCode TSTextLogObjectDestroy(TSTextLogObject the_object) void TSTextLogObjectHeaderSet(TSTextLogObject the_object, const char * header) TSReturnCode TSTextLogObjectRollingEnabledSet(TSTextLogObject the_object, int rolling_enabled) void TSTextLogObjectRollingIntervalSecSet(TSTextLogObject the_object, int rolling_interval_sec) void TSTextLogObjectRollingOffsetHrSet(TSTextLogObject the_object, int rolling_offset_hr) void TSTextLogObjectRollingSizeMbSet(TSTextLogObject the_object, int rolling_size_mb)
Description
TSTextLogObjectRollingEnabledSet() sets the log rolling mode for the given object. This API must be used once the object is created and before writing into logs. The :arg:‘rolling_enabled‘ argument must be a valid :ts:cv:‘proxy.conﬁg.log.rolling_enabled‘ values. If TSTextLogObjectRollingEnabledSet() is never called, the log object takes it’s log rolling mode from the global :ts:cv:‘proxy.conﬁg.log.rolling_enabled‘ setting. It’s also important to call TSTextLogObjectRollingEnabledSet() before any of the other APIs that modiﬁes the rolling object. This is due to the fact that this API dictates which rolling mode should be used, and therefore affects further modiﬁcations of the logging object.
See Also
TSAPI(3ts)
TSThreadCreate
Synopsis
#include  TSThread TSThreadCreate(TSThreadFunc func, void * data)
Description
TSThreadDestroy
Synopsis
#include 
456 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 void TSThreadDestroy(TSThread thread)
Description
TSThreadInit
Synopsis
#include  TSThread TSThreadInit(void)
Description
TSThreadSelf
Synopsis
#include  TSThread TSThreadSelf(void)
Description
TSTrafﬁcServerVersionGet
Return Trafﬁc Server version information.
Synopsis
#include  const char * TSTrafficServerVersionGet(void) int TSTrafficServerVersionGetMajor(void) int TSTrafficServerVersionGetMinor(void) int TSTrafficServerVersionGetPatch(void)
Description
TSTrafficServerVersionGet() returns a pointer to a string of characters that indicates the Trafﬁc Server release version. This string must not be modiﬁed. The other APIs return an integer from the relevant component of the version number string.
4.9. API Reference 457 Apache-Trafﬁcserver-Server Documentation, latest
Example
#include  #include  int check_ts_version() { const char *ts_version = TSTrafficServerVersionGet(); int result = 0;
 if (ts_version) { int major_ts_version = 0; int minor_ts_version = 0; int patch_ts_version = 0;
 if (sscanf(ts_version, "%d.%d.%d", &major_ts_version, &minor_ts_version, &patch_ts_version) != 3) { return 0; }
/* We need at least Traffic Server 3.0 */ if (major_ts_version >= 3) { result = 1; } }
 return result; } void TSPluginInit (int argc, const char *argv[]) { TSPluginRegistrationInfo info; info.plugin_name = "hello-world"; info.vendor_name = "MyCompany"; info.support_email = "[email protected]";
 if (TSPluginRegister(&info) != TS_SUCCESS) { TSError("Plugin registration failed. 0); }
 if (!check_ts_version()) { TSError("Plugin requires Traffic Server 3.0 or later0); return; }
TSDebug("debug-hello", "Hello World!0); }
See Also
TSAPI(3ts)
458 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSTransformCreate
Synopsis
#include  TSVConn TSTransformCreate(TSEventFunc event_funcp, TSHttpTxn txnp)
Description
TSTransformOutputVConnGet
Synopsis
#include  TSVConn TSTransformOutputVConnGet(TSVConn connp)
Description
TSAPI Types
Synopsis
#include  #include 
Description
The Apache Trafﬁc Server API provides large number of types. Many of them are speciﬁc to a particular API function or function group, but others are used more widely. Those are described on this page. ink_hrtime TSAction TSCacheKey TSConfig TSConfigDestroyFunc TSCont An opaque type that represents a Trafﬁc Server continuation. TSEvent TSEvents are sent to continuations when they are called back. The TSEvent provides the continuation’s handler function with information about the callback. Based on the event it receives, the handler function can decide what to do. TSEventFunc TSFile TSHostLookupResult
4.9. API Reference 459 Apache-Trafﬁcserver-Server Documentation, latest
TSHRTime “High Resolution Time” A 64 bit time value, measured in nanoseconds. TSHttpHookID An enumeration that identiﬁes a speciﬁc type of hook for HTTP transactions. TSHttpParser TSHttpSsn An opaque type that represents a Trafﬁc Server session. TSHttpStatus This set of enums represents possible return values from TSHttpHdrStatusGet(), which retrieves the status code from an HTTP response header (TSHttpHdrStatusGet() retrieves status codes only from headers of type TS_HTTP_TYPE_RESPONSE). You can also set the TSHttpStatus of a response header using TSHttpHdrStatusSet(). TSHttpTxn An opaque type that represents a Trafﬁc Server HTTP transaction. TSHttpType This set of enums represents the possible HTTP types that can be assigned to an HTTP header. When a header is created with TSHttpHdrCreate(), it is automatically assigned a type of TS_HTTP_TYPE_UNKNOWN. You can modify the HTTP type ONCE after it the header is created, us- ing TSHttpHdrTypeSet(). After setting the HTTP type once, you cannot set it again. Use TSHttpHdrTypeGet() to obtain the TSHttpType of an HTTP header. TSIOBuffer TSIOBufferBlock TSIOBufferReader TSIOBufferSizeIndex TSLifecycleHookID An enumeration that identiﬁes a life cycle hook. TSMBuffer TSMgmtCounter TSMgmtFloat The type used internally for a ﬂoating point value. This corresponds to the value :const:‘TS_RECORDDATATYPE_FLOAT‘ for TSRecordDataType. TSMgmtInt The type used internally for an integer. This corresponds to the value :const:‘TS_RECORDDATATYPE_INT‘ for TSRecordDataType. TSMgmtString TSMimeParser TSMLoc TSMutex TSParseResult This set of enums are possible values returned by TSHttpHdrParseReq() and TSHttpHdrParseResp().
460 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSPluginRegistrationInfo The following struct is used by TSPluginRegister(). It stores registration information about the plugin. TSRecordDataType An enumeration that speciﬁes the type of a value in an internal data structure that is accessible via the API. TSRemapInterface TSRemapRequestInfo TSRemapStatus TSReturnCode An indicator of the results of an API call. A value of :const:‘TS_SUCCESS‘ means the call was successful. Any other value indicates a failure and is speciﬁc to the API call. TSSDKVersion Starting 2.0, SDK now follows same versioning as Trafﬁc Server. TSServerState TSTextLogObject This type represents a custom log ﬁle that you create with TSTextLogObjectCreate(). Your plugin writes entries into this log ﬁle using TSTextLogObjectWrite(). TSThread TSThreadFunc TSThreadPool TSVConn TSVIO
TSUrlCreate
Trafﬁc Server URL object construction API.
Synopsis
#include  TSReturnCode TSUrlCreate(TSMBuffer bufp, TSMLoc * locp) TSReturnCode TSUrlClone(TSMBuffer dest_bufp, TSMBuffer src_bufp, TSMLoc src_url, TSMLoc * locp) TSReturnCode TSUrlCopy(TSMBuffer dest_bufp, TSMLoc dest_url, TSMBuffer src_bufp, TSMLoc src_url) TSParseResult TSUrlParse(TSMBuffer bufp, TSMLoc offset, const char ** start, const char * end)
Description
The URL data structure is a parsed version of a standard internet URL. The Trafﬁc Server URL API provides access to URL data stored in marshal buffers. The URL functions can create, copy, retrieve or delete entire URLs, and retrieve or modify parts of URLs, such as their host, port or scheme information.
4.9. API Reference 461 Apache-Trafﬁcserver-Server Documentation, latest
TSUrlCreate() creates a new URL within the marshal buffer :arg:‘bufp‘. Release the resulting handle with a call to TSHandleMLocRelease(). TSUrlClone() copies the contents of the URL at location :arg:‘src_url‘ within the marshal buffer :arg:‘src_bufp‘ to a location within the marshal buffer :arg:‘dest_bufp‘. Release the returned handle with a call to TSHandleMLocRelease(). TSUrlCopy() copies the contents of the URL at location :arg:‘src_url‘ within the marshal buffer :arg:‘src_bufp‘ to the location :arg:‘dest_url‘ within the marshal buffer dest_bufp. TSUrlCopy() works correctly even if :arg:‘src_bufp‘ and :arg:‘dest_bufp‘ point to different marshal buffers. It is important for the destination URL (its marshal buffer and TSMLoc) to have been created before copying into it. TSUrlParse() parses a URL. The :arg:‘start‘ pointer is both an input and an output parameter and marks the start of the URL to be parsed. After a successful parse, the :arg:‘start‘ pointer equals the :arg:‘end‘ pointer. The :arg:‘end‘ pointer must be one byte after the last character you want to parse. The URL parsing routine assumes that everything between :arg:‘start‘ and :arg:‘end‘ is part of the URL. It is up to higher level parsing routines, such as TSHttpHdrParseReq(), to determine the actual end of the URL.
Return Values
The TSUrlParse() function returns a TSParseResult, where TS_PARSE_ERROR indicates an error. Suc- cess is indicated by one of TS_PARSE_DONE, TS_PARSE_OK or TS_PARSE_CONT. The other APIs all return a TSReturnCode, indicating success (TS_SUCCESS) or failure (TS_ERROR) of the operation.
See Also
TSAPI(3ts), TSMBufferCreate(3ts), TSUrlHostGet(3ts), TSUrlHostSet(3ts), TSUrlStringGet(3ts), TSUrlPercentEncode(3ts)
TSUrlDestroy
Synopsis
#include  TSReturnCode TSUrlDestroy(TSMBuffer bufp, TSMLoc offset)
Description
Destroys the URL located at :arg:‘url_loc‘ within the marshal buffer :arg:‘bufp‘. Do not forget to release the TSMLoc url_loc with a call to TSHandleMLocRelease().
Deprecated There is no reason to destroy the URL, just release the marshal buffers.
Should be removed for v5.0.0
462 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSUrlFtpTypeGet
Synopsis
#include  int TSUrlFtpTypeGet(TSMBuffer bufp, TSMLoc offset)
Description
Retrieves the FTP type of the URL located at :arg:‘url_loc‘ within :arg:‘bufp‘.
TSUrlFtpTypeSet
Synopsis
#include  TSReturnCode TSUrlFtpTypeSet(TSMBuffer bufp, TSMLoc offset, int type)
Description
Sets the FTP type portion of the URL located at :arg:‘url_loc‘ within :arg:‘bufp‘ to the value :arg:‘type‘.
TSUrlHostGet
Trafﬁc Server URL component retrieval API.
Synopsis
#include  const char * TSUrlHostGet(TSMBuffer bufp, TSMLoc offset, int * length) const char * TSUrlSchemeGet(TSMBuffer bufp, TSMLoc offset, int * length) const char * TSUrlUserGet(TSMBuffer bufp, TSMLoc offset, int * length) const char * TSUrlPasswordGet(TSMBuffer bufp, TSMLoc offset, int* length) int TSUrlPortGet(TSMBuffer bufp, TSMLoc offset) const char * TSUrlPathGet(TSMBuffer bufp, TSMLoc offset, int * length) const char * TSUrlHttpQueryGet(TSMBuffer bufp, TSMLoc offset, int * length) const char * TSUrlHttpParamsGet(TSMBuffer bufp, TSMLoc offset, int * length) const char * TSUrlHttpFragmentGet(TSMBuffer bufp, TSMLoc offset, int * length)
4.9. API Reference 463 Apache-Trafﬁcserver-Server Documentation, latest
Description
The URL data structure is a parsed version of a standard internet URL. The Trafﬁc Server URL API provides access to URL data stored in marshal buffers. The URL functions can create, copy, retrieve or delete entire URLs, and retrieve or modify parts of URLs, such as their host, port or scheme information. TSUrlSchemeGet(), TSUrlUserGet(), TSUrlPasswordGet(), TSUrlHostGet(), TSUrlHttpParamsGet(), TSUrlHttpQueryGet() and TSUrlHttpFragmentGet() each retrieve an internal pointer to the speciﬁed portion of the URL from the marshall buffer :arg:‘bufp‘. The length of the returned string is placed in :arg:‘length‘ and a pointer to the URL portion is returned. TSUrlPortGet() retrieves the port number portion of the URL located at :arg:‘offset‘ within the marshal buffer :arg:‘bufp‘. If there is no explicit port number in the URL, a canonicalized valued is returned based on the URL scheme.
Return Values
All APIs except TSUrlPortGet() returns a string, which is not guaranteed to be NULL terminated. You must therefore always use the :arg:‘length‘ value to determine the actual length of the returned string. TSUrlPortGet() simply returns the port number as an integer, possibly canonicalized with 80 for HTTP and 443 for HTTPS schemes. If there is neither port nor scheme information available in the URL, 0 is returned.
See Also
TSAPI(3ts), TSUrlCreate(3ts), TSUrlHostSet(3ts), TSUrlStringGet(3ts), TSUrlPercentEncode(3ts)
TSUrlHostSet
Trafﬁc Server URL component manipulation API.
Synopsis
#include  TSReturnCode TSUrlHostSet(TSMBuffer bufp, TSMLoc offset, const char * value, int length) TSReturnCode TSUrlSchemeSet(TSMBuffer bufp, TSMLoc offset, const char * value, int length) TSReturnCode TSUrlUserSet(TSMBuffer bufp, TSMLoc offset, const char * value, int length) TSReturnCode TSUrlPasswordSet(TSMBuffer bufp, TSMLoc offset, const char * value, int length) TSReturnCode TSUrlPortSet(TSMBuffer bufp, TSMLoc offset, int port) TSReturnCode TSUrlPathSet(TSMBuffer bufp, TSMLoc offset, const char * value, int length) TSReturnCode TSUrlHttpQuerySet(TSMBuffer bufp, TSMLoc offset, const char * value, int length) TSReturnCode TSUrlHttpParamsSet(TSMBuffer bufp, TSMLoc offset, const char * value, int length) TSReturnCode TSUrlHttpFragmentSet(TSMBuffer bufp, TSMLoc offset, const char * value, int length)
464 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
The URL data structure is a parsed version of a standard internet URL. The Trafﬁc Server URL API provides access to URL data stored in marshal buffers. The URL functions can create, copy, retrieve or delete entire URLs, and retrieve or modify parts of URLs, such as their host, port or scheme information. TSUrlSchemeSet(), TSUrlUserSet(), TSUrlPasswordSet(), TSUrlHostSet(), TSUrlHttpParamsSet(), TSUrlHttpQuerySet() and TSUrlHttpFragmentSet() each set the speciﬁed portion of the URL located at offset within the marshal buffer :arg:‘bufp‘ to the string value. If :arg:‘length‘ is -1 then these functions assume that value is NULL-terminated. Otherwise, the length of the :arg:‘string‘ value is taken to be the value of :arg:‘length‘. These functions copy the string to within :arg:‘bufp‘, so it can be subsequently modiﬁed or deleted. TSUrlPortSet() sets the port number portion of the URL located at :arg:‘offset‘ within the marshal buffer :arg:‘bufp‘ to the value port. Normal canonicalization based on the URL scheme still applies.
Return Values
All these APIs returns a TSReturnCode, indicating success (TS_SUCCESS) or failure (TS_ERROR) of the opera- tion.
See Also
TSAPI(3ts), TSUrlCreate(3ts), TSUrlHostGet(3ts), TSUrlStringGet(3ts), TSUrlPercentEncode(3ts)
TSUrlPercentEncode
Trafﬁc Server URL percent encoding API.
Synopsis
#include  TSReturnCode TSUrlPercentEncode(TSMBuffer bufp, TSMLoc offset, char * dst, size_t dst_size, size_t * length, const unsigned char * map) TSReturnCode TSStringPercentEncode(const char * str, int str_len, char * dst, size_t dst_size, size_t * length, const unsigned char * map) TSReturnCode TSStringPercentDecode(const char * str, size_t str_len, char * dst, size_t dst_size, size_t * length)
Description
The URL data structure is a parsed version of a standard internet URL. The Trafﬁc Server URL API provides access to URL data stored in marshal buffers. The URL functions can create, copy, retrieve or delete entire URLs, and retrieve or modify parts of URLs, such as their host, port or scheme information. TSUrlPercentEncode() performs percent-encoding of the URL object, storing the new string in the :arg:‘dst‘ buffer. The :arg:‘length‘ parameter will be set to the new (encoded) string length, or 0 if the encoding failed.
4.9. API Reference 465 Apache-Trafﬁcserver-Server Documentation, latest
TSStringPercentEncode() is similar but operates on a string. If the optional :arg:‘map‘ parameter is pro- vided (not NULL) , it should be a map of characters to encode. TSStringPercentDecode() perform percent-decoding of the string in the :arg:‘str‘ buffer, writing to the :arg:‘dst‘ buffer. The source and destination can be the same, in which case they overwrite. The decoded string is always guaranteed to be no longer than the source string.
Return Values
All these APIs returns a TSReturnCode, indicating success (TS_SUCCESS) or failure (TS_ERROR) of the opera- tion.
See Also
TSAPI(3ts), TSUrlCreate(3ts), TSUrlHostGet(3ts), TSUrlHostSet(3ts), TSUrlStringGet(3ts)
TSUrlStringGet
Trafﬁc Server URL string representations API.
Synopsis
#include  char * TSUrlStringGet(TSMBuffer bufp, TSMLoc offset, int * length) int TSUrlLengthGet(TSMBuffer bufp, TSMLoc offset) void TSUrlPrint(TSMBuffer bufp, TSMLoc offset, TSIOBuffer iobufp)
Description
The URL data structure is a parsed version of a standard internet URL. The Trafﬁc Server URL API provides access to URL data stored in marshal buffers. The URL functions can create, copy, retrieve or delete entire URLs, and retrieve or modify parts of URLs, such as their host, port or scheme information. TSUrlStringGet() constructs a string representation of the URL located at :arg:‘offset‘ within the marshal buffer :arg:‘bufp‘. TSUrlStringGet() stores the length of the allocated string in the parameter :arg:‘length‘. This is the same length that TSUrlLengthGet() returns. The returned string is allocated by a call to TSmalloc() and must be freed by a call to TSfree(). If length is NULL then no attempt is made to de-reference it. TSUrlLengthGet() calculates the length of the URL located at :arg:‘offset‘ within the marshal buffer bufp as if it were returned as a string. This length will be the same as the length returned by TSUrlStringGet(). TSUrlPrint() formats a URL stored in an TSMBuffer to an TSIOBuffer.
See Also
TSAPI(3ts), TSmalloc(3ts), TSUrlCreate(3ts), TSUrlHostGet(3ts), TSUrlHostSet(3ts), TSUrlPercentEncode(3ts)
466 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSVConnAbort
Synopsis
#include  void TSVConnAbort(TSVConn connp, int error)
Description
TSVConnCacheObjectSizeGet
Synopsis
#include  int64_t TSVConnCacheObjectSizeGet(TSVConn connp)
Description
TSVConnClose
Synopsis
#include  void TSVConnClose(TSVConn connp)
Description
TSVConnClosedGet
Synopsis
#include  int TSVConnClosedGet(TSVConn connp)
Description
TSVConnFdCreate
Create a TSVConn from a socket.
Synopsis
#include  TSVConn TSVConnFdCreate(int fd)
4.9. API Reference 467 Apache-Trafﬁcserver-Server Documentation, latest
Description
TSVConnFdCreate() accepts a network socket :arg:‘fd‘ and returns a new TSVConn constructed from the socket. The socket descriptor must be an already connected socket. It will be placed into non-blocking mode.
Return Values
On success, the returned TSVConn object owns the socket and the caller must not close it. If TSVConnFdCreate() fails, NULL is returned, the socket is unchanged and the caller must close it.
Examples
The example below is excerpted from example/intercept/intercept.cc in the Trafﬁc Server source distribution. It demonstrates how to use TSVConnFdCreate() to construct a TSVConn from a connected socket.
See Also
TSAPI(3ts)
TSVConnIsSsl
Synopsis
#include  int TSVConnIsSsl(TSVConn svc)
Description
Determines whether the connection associated with :arg:‘svc‘ is being processed as an SSL connection. Returns 1 if it is being processed as SSL and 0 otherwise.
TSVConnRead
Synopsis
#include  TSVIO TSVConnRead(TSVConn connp, TSCont contp, TSIOBuffer bufp, int64_t nbytes)
Description
TSVConnReadVIOGet
Synopsis
#include 
468 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
TSVIO TSVConnReadVIOGet(TSVConn connp)
Description
TSVConnReenable
Synopsis
#include  void TSSslVConnReenable(TSVConn svc)
Description
Reenable the SSL connection :arg:‘svc‘. If a plugin hook is called, ATS processing on that connnection will not resume until this is invoked for that connection. If the server is running OpenSSL 1.0.1 with the appropraite patch installed or it is running OpenSSL 1.0.2, the plugin writer can pause SSL handshake processing by not reenabling the connection. Without the OpenSSL patch or running an OpenSSL versions older than 1.0.2, the handshake processing in SSL_accept will not be stopped even if the SNI callback does not reenable the connection. Additional processing could reenable the virtual connection causing the SSL_accept to be called again to complete the handshake exchange. In the case of a blind tunnel conversion, the SSL handshake will never be completed by Trafﬁc Server. This call does appropriate locking and scheduling, so it is safe to call from another thread.
TSVConnShutdown
Synopsis
#include  void TSVConnShutdown(TSVConn connp, int read, int write)
Description
TSVConnSslConnectionGet
Synopsis
#include  TSSslVConnection TSVConnSslConnectionGet(TSVConn svc)
Description
Get the SSL (per connection) object from the SSl connection :arg:‘svc‘.
4.9. API Reference 469 Apache-Trafﬁcserver-Server Documentation, latest
Types
TSSslVConnection The SSL (per connection) object. This is an opaque type that can be cast to the appropriate type (SSL * for the OpenSSL library).
TSVConnTunnel
Synopsis
#include  TSReturnCode TSVConnTunnel(TSVConn svc)
Description
Set the SSL connection :arg:‘svc‘ to convert to a blind tunnel. Can be called from TS_VCONN_PRE_ACCEPT_HOOK or TS_SSL_SNI_HOOK/TS_SSL_CERT_HOOK. For this to work from the TS_SSL_SNI_HOOK or TS_SSL_CERT_HOOK, either the server must be running OpenSSL 1.0.2 or a version of OpenSSL 1.0.1 with the appropriate patch installed.
TSVConnWrite
Synopsis
#include  TSVIO TSVConnWrite(TSVConn connp, TSCont contp, TSIOBufferReader readerp, int64_t nbytes)
Description
TSVConnWriteVIOGet
Synopsis
#include  TSVIO TSVConnWriteVIOGet(TSVConn connp)
Description
TSVIOBufferGet
Synopsis
#include  TSIOBuffer TSVIOBufferGet(TSVIO viop)
470 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
TSVIOContGet
Synopsis
#include  TSCont TSVIOContGet(TSVIO viop)
Description
TSVIOMutexGet
Synopsis
#include  TSMutex TSVIOMutexGet(TSVIO viop)
Description
TSVIONBytesGet
Synopsis
#include  int64_t TSVIONBytesGet(TSVIO viop)
Description
TSVIONBytesSet
Synopsis
#include  void TSVIONBytesSet(TSVIO viop, int64_t nbytes)
Description
TSVIONDoneGet
Synopsis
#include  int64_t TSVIONDoneGet(TSVIO viop)
4.9. API Reference 471 Apache-Trafﬁcserver-Server Documentation, latest
Description
TSVIONDoneSet
Synopsis
#include  void TSVIONDoneSet(TSVIO viop, int64_t ndone)
Description
TSVIONTodoGet
Synopsis
#include  int64_t TSVIONTodoGet(TSVIO viop)
Description
TSVIOReaderGet
Synopsis
#include  TSIOBufferReader TSVIOReaderGet(TSVIO viop)
Description
TSVIOReenable
Synopsis
#include  void TSVIOReenable(TSVIO viop)
Description
TSVIOVConnGet
Synopsis
#include  TSVConn TSVIOVConnGet(TSVIO viop)
472 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Description
TSfclose
Synopsis
#include  void TSfclose(TSFile ﬁlep)
Description
Closes the ﬁle to which :arg:‘ﬁlep‘ points and frees the data structures and buffers associated with it. If the ﬁle was opened for writing, any pending data is ﬂushed.
TSfﬂush
Synopsis
#include  void TSfflush(TSFile ﬁlep)
Description
Flushes pending data that has been buffered up in memory from previous calls to TSfwrite().
TSfgets
Synopsis
#include  char* TSfgets(TSFile ﬁlep, char * buf, size_t length)
Description
Reads a line from the ﬁle pointed to by :arg:‘ﬁlep‘ into the buffer :arg:‘buf‘. Lines are terminated by a line feed character, ‘ ‘. The line placed in the buffer includes the line feed character and is terminated with a NULL. If the line is longer than length bytes then only the ﬁrst length-minus-1 bytes are placed in :arg:‘buf‘.
TSfopen
Synopsis
#include 
4.9. API Reference 473 Apache-Trafﬁcserver-Server Documentation, latest
TSFile TSfopen(const char * ﬁlename, const char * mode)
Description
Opens a ﬁle for reading or writing and returns a descriptor for accessing the ﬁle. The current implementation cannot open a ﬁle for both reading and writing. See TSfopen Family for examples.
TSfread
Synopsis
#include  size_t TSfread(TSFile ﬁlep, void * buf, size_t length)
Description
Attempts to read :arg:‘length‘ bytes of data from the ﬁle pointed to by :arg:‘ﬁlep‘ into the buffer :arg:‘buf‘.
TSfwrite
Synopsis
#include  size_t TSfwrite(TSFile ﬁlep, const void * buf, size_t length)
Description
Attempts to write :arg:‘length‘ bytes of data from the buffer :arg:‘buf‘ to the ﬁle :arg:‘ﬁlep‘. Make sure that :arg:‘ﬁlep‘ is open for writing. You might want to check the number of bytes written (TSfwrite() returns this value) against the value of :arg:‘length‘. If it is less, there might be insufﬁcient space on disk, for example.
TSmalloc
Trafﬁc Server memory allocation API.
Synopsis
#include  void * TSmalloc(size_t size, const char * path) void * TSrealloc(void * ptr, size_t size, const char * path) char * TSstrdup(const char * str) char * TSstrndup(const char * str, size_t size) size_t TSstrlcpy(char * dst, const char * src, size_t size)
474 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 size_t TSstrlcat(char * dst, const char * src, size_t size) void TSfree(void * ptr)
Description
Trafﬁc Server provides a number of routines for allocating and freeing memory. These routines correspond to similar routines in the C library. For example, TSrealloc() behaves like the C library routine realloc(). There are two reasons to use the routines provided by Trafﬁc Server. The ﬁrst is portability. The Trafﬁc Server API routines behave the same on all of Trafﬁc Servers supported platforms. For example, realloc() does not accept an argument of NULL on some platforms. The second reason is that the Trafﬁc Server routines actually track the memory allocations by ﬁle and line number. This tracking is very efﬁcient, is always turned on, and is useful for tracking down memory leaks. TSmalloc() returns a pointer to size bytes of memory allocated from the heap. Trafﬁc Server uses TSmalloc() internally for memory allocations. Always use TSfree() to release memory allocated by TSmalloc(); do not use free(). TSstrdup() returns a pointer to a new string that is a duplicate of the string pointed to by str. The memory for the new string is allocated using TSmalloc() and should be freed by a call to TSfree(). TSstrndup() returns a pointer to a new string that is a duplicate of the string pointed to by str and size bytes long. The new string will be NUL-terminated. This API is very useful for transforming non NUL-terminated string values returned by APIs such as TSMimeHdrFieldValueStringGet() into NUL-terminated string values. The memory for the new string is allocated using TSmalloc() and should be freed by a call to TSfree(). TSstrlcpy() copies up to size - 1 characters from the NUL-terminated string src to dst, NUL-terminating the result. TSstrlcat() appends the NUL-terminated string src to the end of dst. It will append at most size - strlen(dst) - 1 bytes, NUL-terminating the result. TSfree() releases the memory allocated by TSmalloc() or TSrealloc(). If ptr is NULL, TSfree() does no operation.
See also
TSAPI(3ts)
Continuous Integration
Documentation
Conventions
The conventions detailed in this chapter should be followed when modifying the project documentation to maintain readability and consistency.
Typographic
Italic Used to introduce new terms. Italics used solely for emphasis should be avoided. When introducing new terms, only the ﬁrst use should be set in italics, and its introduction should be followed with a deﬁnition or description of the term.
4.10. Continuous Integration 475 Apache-Trafﬁcserver-Server Documentation, latest
Example:
The|ATS| object storage is based on a *cyclone buffer* architecture. Cyclone buffers are most simply described as a fixed size, but continually looping block of storage updated by a single writer process, wherein the writer continually reclaims the oldest allocations for the most recent object updates.
Bold Bold typesetting is to be reserved for section, table, and glossary headings and should be avoided in paragraph copy. Monospace Used to indicate ﬁlesystem paths, variable names, language keywords and functions, and command out- put. Note that in the case of variables and functions, whenever possible references to their documentation should be used in place of simple monospace markup. Example:
Documentation source files for |TS| are located in the ``doc/`` directory of the project source tree.
Bracketed Monospace Used to indicate, within command or source code examples, variables for which the reader should substitute a value. Example:
To examine a performance statistic of a running |TS| instance, you may run the comand ``traffic_line -r ``, replacing ```` with the statistic you wish to examine.
Ellipsis Used to indicate the omission of irrelevant or unimportant information. May also be used to separate matter to be treated in detail elsewhere.
Layout Styles
Block Content
Notes Use of .. note:: blocks should be sparing. In most rendered forms, the note block will appear quite prominently and draw the readers’ eyes away from the surrounding copy. It is therefore advisable that the note itself provide enough context and detail to be read on its own, without a full reading of the surrounding copy. Important Notes The use of .. important:: callout blocks should be limited only to those situations in which critical information needs to be prominently displayed. Suitable use would include noting that resizing a cache voiume will result in Trafﬁc Server resetting the cache contents to empty when the service is started. This is information that may not be obvious, or safe to assume, for the reader but which can signiﬁcantly (and negatively) impact the use and performance of Trafﬁc Server in their environment. Important note blocks should not be used for behavior or actions which generally do not have potential negative side effects. Sidebars The use of .. sidebar:: blocks in Trafﬁc Server documentation should be avoided, and note blocks favored in their place. Code Samples Content should be set within .. code:: blocks whenever a full line or multiple lines of source code are being included in the documentation, or when example shell or network commands are being demonstrated. Blank lines may be used to separate lines of code within the block for readability, and all normal indentation practices should be followed. No additional markup should be used within the content block. Deﬁnition Lists Deﬁnition lists may be used in multiple ways, not just for the term listings in the glossary section. They may be used to provide individual detailed treatment for a list of function or command arguments or where any series of terms need to be explained outside of the formal glossary.
476 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Ordered Lists Explicityly numbered ordered lists should be avoided. reStructuredText provides two methods of marking up ordered, numbered lists, and the automatic numbering form should be used in all cases where surrounding paragraphs do not need to reference individual list entries.
Tables
Tabular content may be used in any situation where a selection of items, terms, options, formats, etc. are being compared or where a grouping of the same are being presented alongside a small number of attributes which do not require detailed expositions. The Trafﬁc Server project documentation supplies a custom styling override which causes cell contents to wrap within wide tables whenever possible in cases where it is necessary to prevent the content from overﬂowing into the margin. If, however, the cell content cannot be wrapped because there are no breaking spaces present (for example, a long variable name containing only letters, periods, underscores, dashes, and so no, but no whitespace), the table may still require overﬂowing into the page margin. Whenever possible, please try to avoid the use of tables when presenting information that will lead to this, as it greatly hampers readibility on smaller screens, especially tablets and mobile devices. Alternatives such as a deﬁnition list may be better suited to the content. Tables may be marked up using any of the reStructuredText styles, though it is generally easiest to maintain those using the simple table markup style.
Structural
Common Deﬁnitions
The Trafﬁc Server project documentation maintains a common deﬁnitions, abbreviations, and shortcut listing in the ﬁle doc/common.defs. This ﬁle should be included by all reStructuredText source ﬁles after the standard project copyright notice. The ﬁle should always be included using a relative path from the current ﬁle’s location. Any commonly or repeatedly used abbreviations, especialy those of product, company, or person names, should be added to the deﬁnitions ﬁle as useful to avoid repetitive typing and ensure accurate spellings or legal usage.
Tables of Content
Any chapters of non-trivial scope should begin with a table of contents on the chapter index. The :depth: option for the table of contents may be set to a value appropriate to the amount of content in the chapter sections. A depth of 2 will generally provide a balance between usefully describing the contents of the chapter without overwhelming a reader scanning for topics relevant to their needs or interests.
Sections and Headings
Each chapter section should be located in a separate reStructuredText ﬁle. Each ﬁle should contain the standard project copyright notice ﬁrst, followed by a unique reference marker for the section. While reStructuredText itself does not deﬁne a ﬁxed ordering of section markers, Trafﬁc Server documentation source ﬁles should use the same set of single line section markings, proceeding through the section levels without skipping. For consistency, the following section line markers should be used in order:
Top Level *********
4.11. Documentation 477 Apache-Trafﬁcserver-Server Documentation, latest
Second Level ======
Third Level ------
Fourth Level ~~~~~~~~~~~~
Any section ﬁle which reaches or has need to exceed the fourth level style of section line markings is an excellent candidate for breaking into several smaller, and ideally more focused, reStructuredText source ﬁles and referenced by an index with its own table of contents.
Footnotes and Endnotes
Both footnotes and endnotes should be avoided. The Trafﬁc Server documentation is intended primarily for online viewing and the positioning of footnotes in the rendered output is not always intuitive. In cases where a footnote might have been appropriate in print-oriented material for referencing an external resource, that reference is more ideally integrated as a standard reStructuredText reference. For more descriptive content that might have been included as a footnote, it is less disruptive and more useful to choose between reformullating the text to simply include the additional wording, or consider the use of an inline note block.
Grammatical
Structure
The Apache Trafﬁc Server™ documentation is presented as a single manual, sub-divided into the following sections: Preface The introductory section to the manual as a whole, with a brief exposition on Trafﬁc Server and its general role in a network and application infrastructure. The typographic conventions used in the manual should be covered here to ease readers’ comprehension of the following sections, as well as pointers to additional resources available outside of the manual. Getting Started Aimed at administrators and developers new to Trafﬁc Server who wish to install and conﬁgure a Trafﬁc Server instance in the least amount of time necessary, without delving into the full featureset or the internals of the Trafﬁc Server architecture. Administrator’s Guide Provides in-depth coverage of all Trafﬁc Server features and conﬁgurations for use by ad- ministrators responsible for installing, conﬁguring, and maintaining Trafﬁc Server instances. Troubleshooting, performance tuning, and plugin usage should all be covered in this section of the manual. Developer’s Guide Documentation of internal Trafﬁc Server architecture as well as guides on plugin development and debugging are presented in this section. Appendices Glossary, common references to both administrator and developer guides (such as status code tables and command line utility usage), frequently asked questions, and errata should all be located in this ﬁnal section of the manual. Within each top-level section, a short introduction should be provided to the content of that section, as well as a brief note on the intended audience. A very high level treatment of each chapter should be presented to allow readers to more easily locate the most relevant chapters for their current needs.
478 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Building the Documentation
All documentation and related ﬁles are located in the source tree under the doc/ directory. Makeﬁles are generated automatiically by the main conﬁgure script. For simplicity’s sake, it is recommended that contributors new to the documentation make use of the included Vagrant conﬁgurations, as these will take care of providing all dependencies. Please refer to the Using Vagrant to Test Trafﬁc Server chapter for complete details on using Vagrant to build and test the Trafﬁc Server source tree. With a conﬁgured source tree, building the documentation requires only the invocation make html from within doc/. For repeated builds while working on the documentation, it is advisable to clean out the built and intermediate ﬁles each time by running the following instead (again, from within the doc/ directory of the Trafﬁc Server source tree):
 make clean&& make&& make html
This will ensure that make doesn’t inadvertantly skip the regeneration of any targets. To view the built documentation, you may point any browser to the directory doc/docbuild/html/. If you are building the documentation on your local machine, you may access the HTML documentation ﬁles directly without the need for a full-ﬂedged web server, as all necessary resources (CSS, Javascript, and images) are referenced using relative paths and there are no server-side scripts necessary to render the documentation. reStructuredText and Sphinx
The Trafﬁc Server documentation source ﬁles are written using reStructuredText (RST) markup. This is a plain-text format intended to be easily readable both in its source and ﬁnal rendered forms. While not entirely dissimilar to other plain text markup formats like Markdown, RST provides additional functionality for deﬁning internal links between documents, producing tables of contents, and indices. Additionally, RST works in hand with Sphinx for providing advanced features such as markup domains, of which this documenentation makes extensive use. Both reStructuredText and Sphinx have ofﬁcial documentation resources which detail their full capabilities and all markup options, as well as methods of extending them with custom markup. This section will explain the most common markup features used, but the following resources may be consulted for further details. • reStructuredText Markup Speciﬁcation • Sphinx Documentation Generator • RST Primary by Sphinx
Trafﬁc Server Speciﬁc Markup
This section covers markup used speciﬁc to the Trafﬁc Server documentation and custom domains provided by local extensions.
Types
Data types are documented using the standard :c:type: markup provided by Sphinx. Types provided by the C API should be documented in doc/developer-guide/api/types/
Constants
Functions
4.11. Documentation 479 Apache-Trafﬁcserver-Server Documentation, latest
Custom Domains
The Trafﬁc Server documentation provides a Sphinx extension, located at doc/ext/traffic-server.py which deﬁnes custom domains for various purposes. Those domains and their usage are documented here.
Conﬁguration Variables
Conﬁguration variables are documented with the :ts:cv: domain, which takes three required arguments (the scope, which is the literal string CONFIG, the variable name, and the data type) and a fourth optional argument (the default value).
:ts:cv:
Deﬁnition
Scope The scope of the variable. For conﬁguration variables, this will always be the literal string CONFIG. Variable Name The full and exact conﬁguration variable name. Data Type Indicates the data type of the variable and must be one of the following literal strings: INT Any integer value. Values may optionally be expressed with a binary order of magnitude sufﬁx; e.g. K for value * 1024, M for value * 1024^2, G for value * 1024^3, or T for value * 1024^4. FLOAT Any ﬂoating point value. STRING Any alphanumeric string. Value The default value of the conﬁguration variable. It is preferable to not use any order of magnitude sufﬁx, as Trafﬁc Server will rewrite its conﬁguration ﬁles under varioua circumstances, and when doing so it does not maintain those sufﬁxes.
Options
The domain for conﬁguration variables takes only a single option at this time. deprecated A simple ﬂag option which, when attached to a conﬁguration variable, is used to indicate that the variable has been deprecated and should no longer be used or relied upon, as it may be removed at any time by future releases.
References
References to conﬁguration variables from elsewhere in the documentation should be made using the standard domain reference markup:
:ts:cv:`full.variable.name`
Statistics
Trafﬁc Server statistics are documented using the domain :ts:stat:. The domain takes three required arguments (collection, name, data type) and an option fourth (an example value).
480 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
:ts:stat:
Deﬁnition
Collection The key name of the collection in the returned JSON data from the Stats Over HTTP Plugin plugin. For most statistics, this is the literal sting global. Required Statistic Name The exact and full name of the statistic. Required Data Type One of the following literal string values: integer, float, boolean, or string. Required Example Value A valid example of the value which may be exported by the statistic. In cases where the description of the statistic makes note of particular features of the values, such as the case of string statistics which may be formatted in speciﬁc ways, providing the optional example is strongly recommended. Optional The statistics domain also supports several options which can provide even more metadata about the statistic. These are currently:
Options type Deﬁnes the type of metric exposed by the statistic. Valid values are: counter Numeric values which only increment based on the accumulation or occurrence of underlying events. Examples include the total number of incoming connections, or the total number of bytes transferred. gauge Used for moment-in-time metrics, such as the current number of Trafﬁc Server processes running, or the current number of connections open to origin servers. flag Indicates that values for this statistic will be an integer with each value indicating a particular state. Most statistics of this type are booleans, where 1 indicates a truth or an on state, and 0 the opposite. derivative Statistics of this type presents values that are calculated, or derived, from other staistics. They do not expose a number or state gathered directly. Typical statistics of this type are representations of a statistic over a given period (e.g. average origin connections per second), ratio or percentage of a statistic as part of a set (e.g. the percentage of total dns lookups which have failed), or any other statistic whose computation depends on the value(s) of one or more other statistics. unit Indicates the units of measurement that should be assumed for the given statistic’s value. introduced May be used to indicate the version of Trafﬁc Server in which the statistic was ﬁrst available. The value of this option should be a valid, human-readable version number string, e.g. 5.3.0. deprecated Used to indicate that the statistic is no longer supported and may be removed in later versions. This option may be used as a simple ﬂag without any given value, or may have a value associated in which case it should be a valid, human-readable version number string for the Trafﬁc Server release which was ﬁrst to deprecate the statistic. ungathered A simple ﬂag option, without any associated values, indicating that while the statistic is included in the output of plugins like Stats Over HTTP Plugin there is no underlying data gathered for the statistic. If a statistic is thus marked, it should be assumed to be invalid or simply unimplemented.
References
To reference a statistic from elsewhere in the documentation, the standard domain reference markup should be used:
4.11. Documentation 481 Apache-Trafﬁcserver-Server Documentation, latest
:ts:stat:`full.statistic.name`
References should not include the collection name, data type, or any other components aside from the statistic name.
Creating New Domains
In the event a new type of object or reference needs to be documented, and none of the existing markup options or domains are appropriate, it is possible to extend reStructuredText and Sphinx by adding custom domains. Each domain may be designed to accept any number of required and optional arguments, as well as any collection of domain options, and each option may be designed to support arbitrary values, restricted (enumerated) values, or to simply act as ﬂags. All custom domain deﬁnitions should be located in doc/ext/traffic-server.py and consist of, at a bare minimum, a domain class deﬁnition and a domain reference class deﬁnition. Sphinx domains are implemented in Python. For this section, we will use the contrived example of creating a domain which permits us to deﬁne and reference a set of variables which are constrained by the following characteristics: 1. Each variable in the domain must be one of known list of data types, which we will limit here to the possibilities of integeer, float, string. 2. Where the data type is not speciﬁed, we can assume it is string. 3. Variables which are numeric in their type may have a range of permissible values. 4. Variables in the domain may still be present and supported in the system, but are planned to be removed in some future release. 5. Every variable is associated with a single URI protocol, though there is no validation performed on the value used to represent the protocol name. As stated, this example is fairly contrived and would not match any particularly likely real-world needs, but it will allow us to demonstrate the full extent of custom domain deﬁnition without needless complexity, reader’s suspension of disbelief permitting. For this chapter’s purpose, we will call this domain simply Variables, and we will construct classes which allow us to document variables thusly:
.. ts:variable:: http_enabled http integer :deprecated:
Enables (any postive, non-zero value) or disables (any zero or negative value) processing of HTTP requests.
And referencing of those variables deﬁned with this domain via:
:ts:variable:`http_enabled`
Deﬁning the Domain
Each domain is deﬁned by a class which inherits from std.Target. Several class attributes are expected, which determine how domain object deﬁnitions are processed. Trafﬁc Server convention is to name each domain’s class in camel case, beginning with TS to prevent any class name collisions with builtin Sphinx classes.
482 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 class TSVariable(std.Target):
We have named the domain’s deﬁning class as TSVariable and inherited from the std.Target class. Given the earlier stated requirements, we need a domain which supports at least two required attributes (a name, of course, and a URI protocol with which it is associated) and a commonly deﬁned, though optional, third attribute (a data type). We’ll deal with the value ranges and deprecation status later. class TSVariable(std.Target): required_arguments=2 optional_arguments=1 final_argument_whitespace= False has_content= True
We’ve now speciﬁed the appropriate number of required and optional arguments, though not what each one happens to be or in what order the required arguments need be written. Additionally, we’ve declared that deﬁnitions using this domain do not permit whitespace in the ﬁnal argument, but deﬁnitions can have a block of text content which follows them and should be associated with the item being deﬁned.
: Permitting whitespace in the ﬁnal argument causes the ﬁnal value of a valid deﬁnition to slurp the remaining content of the deﬁnition. Normally, each argument is separated by whitespace, thus foo bar baz would only be a valid deﬁnition if the domain’s required and optional argument counts added up to exactly three. If the domain deﬁned only two arguments as expected, but sets final_argument_whitespace to True, then the deﬁnition would be valid and the second argument in this case would be bar baz.
Our requirements also state support for optional value ranges, and a ﬂag to indicate whether the variable is being deprecated. These can easily be supported through the option_spec, which allows for options to be tagged on to a domain item, on the lines immediately following its deﬁnition. class TSVariable(std.Target): ... option_spec={ 'deprecated' : rst.directives.flag, 'range' : rst.directives.unchanged }
For our example, deprecated is simply a boolean ﬂag, and range will be an arbitrary string on which we will perform no particular transformation or validation (good behavior will be left up to those documenting their variables with this domain). The rst.directives module may be consulted for a wider range of predeﬁned option types, including the ability to deﬁne your own types which can perform any complexity of validation you may desire to implement. It would be good form to also include a docstring for the class explaining the expected arguments in brief. With that included, our class now looks like: class TSVariable(std.Target): """ Description of a Traffic Server protocol variable.
Required arguments, in order, are:
URI Protocol Variable name
Optional argument is the data type of the variable, with "string" the the default. Possible values are: "string", "integer", and "float".
4.11. Documentation 483 Apache-Trafﬁcserver-Server Documentation, latest
Options supported are:
:deprecated: - A simple flag option indicating whether the variable is slated for removal in future releases.
:range: - A string describing the permissible range of values the variable may contain. """
 option_spec={ 'deprecated' : rst.directives.flag, 'range' : rst.directives.unchanged }
 required_arguments=2 optional_arguments=1 final_argument_whitespace= False has_content= True
Every domain class must also provide a run method, which is called every time an item deﬁnition using the domain is encountered. This method is where all argument and option validations are performed, and where transformation of the deﬁnition into the documentation’s rendered output occurs. The core responsibilities of the run method in a domain class are to populate the domain’s data dictionary, for use by references, as well as to transform the item’s deﬁnition into a document structure suitable for rendering. The default title to be used for references will be constructed in this method, and all arguments and options will be processed. Our variables domain might have the following run method: def run(self): var_name, var_proto= self.arguments[0:2] var_type='string'
 if (len(self.arguments)>2): var_type= self.arguments[2]
# Create a documentation node to use as the parent. node= sphinx.addnodes.desc() node.document= self.state.document node['objtype']='variable'
# Add the signature child node for permalinks. title= sphinx.addnodes.desc_signature(var_name,'') title['ids'].append(nodes.make_id('variable-'+var_name)) title['names'].append(var_name) title['first']= False title['objtype']= node['objtype'] self.add_name(title) title.set_class('ts-variable-title')
 title+= sphinx.addnodes.desc_name(var_name, var_name) node.append(title)
 env.domaindata['ts']['variable'][var_name]= env.docname
# Create table detailing all provided domain options fl= nodes.field_list()
484 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
 if ('deprecated' in self.options): fl.append(self.make_field('Deprecated','Yes'))
 if ('range' in self.options): fl.append(self.make_field('Value range:', self.options['range']))
# Parse any associated block content for the item's description nn= nodes.compound() self.state.nested_parse(self.content, self.content_offset, nn)
# Create an index node so Sphinx will list this variable and its # references in the index section. indexnode= sphinx.addnodes.index(entries=[]) indexnode['entries'].append( ('single', _('%s')% var_name, nodes.make_id(var_name),'') )
 return [ indexnode, node, fl, nn ]
Deﬁning the Domain Reference
Domain reference deﬁnitions are quite simple in comparison to the full domain deﬁnition. As with the domain itself, they are deﬁned by a single class, but inherit from XRefRole instead. There are no attributes necessary, and only a single method, process_link need be deﬁned. For our variables domain references, the class deﬁnition is a very short one. Trafﬁc Server convention is to name the reference class the same as the domain class, but with Ref appended to the name. Thus, the domain class TSVariable is accompanied by a TSVariableRef reference class.
 class TSVariableRef(XRefRole): def process_link(self, env, ref_node, explicit_title_p, title, target): return title, target
The process_link method will receive several arguments, as described below, and should return two values: a string containing the title of the reference, and a hyperlink target to be used for the rendered documentation. The process_link method receives the following arguments: self The reference instance object, as per Python method conventions. env A dictionary object containing the environment of the documentation processor in its state at the time of the reference encounter. ref_node The node object of the reference as encountered in the documentation source. explicit_title_p Contains the text content of the reference’s explicit title overriding, if present in the reference markup. title The processed form of the reference title, which may be the result of domain class transformations or an overriding of the reference title within the reference itself. target The computed target of the reference, suitable for use by Sphinx to construct hyperlinks to the location of the item’s deﬁnition, wherever it may reside in the ﬁnal rendered form of the documentation. In our reference class, we have simply returned the processed title (allowing the documentation to override the vari- able’s name if desired, or defaulting to the domain class’s representation of the variable name in all other cases) and the parser’s computed target.
4.11. Documentation 485 Apache-Trafﬁcserver-Server Documentation, latest
It is recommended to leave the target untouched, however you may choose to perform any transformations you wish on the value of the title, bearing in mind that whatever string is returned will appear verbatim in the rendered documentation everywhere references for this domain are used.
Exporting the Domain
With both the domain itself and references to it now deﬁned, the ﬁnal step is to register those classes as domain and reference handlers in a namespace. This is done for Trafﬁc Server (in its :ts: namespace) quite easily by modifying the TrafficServerDomain class, also located in doc/ext/traffic-server.py. The following dictionaries deﬁned by that class should be updated to include the new domain and reference. In each case, the key used when adding to the dictionary should be the string you wish to use in documentation markup for your new domain. In our example’s case, we will choose variable since it aligns with the Python classes we’ve created above, and their contrived purpose. object_types Used to deﬁne the actual markup string directives Deﬁnes which class is used to implement a given domain. roles Deﬁnes the class used to implement references to a domain. initial_data Used to initialized the dictionary which tracks all encountered instances of each domain. This should always be set to an empty dictionary for each domain. dangling_warnings May be used to provide a default warning if a reference is attempted to a non-existent item for a domain.
Documenting Plugins
Plugins can provide tremendously useful functionality for users of Trafﬁc Server, but it does them little good if there are no instructions on how to use the plugins properly. In the interest of encouraging useful documentation on all the features available to those deploying Trafﬁc Server in their infrastructure, this section lays out a series of guidelines and offers a basic template for adding new entries to the Plugins chapter of the Administrator’s Guide.
Structure
Ideally, every plugin will provide documentation covering the following general topics (preferably in top-level sections named to match):
Introduction
A brief section in which the plugin’s core functionality is explained in a quickly digestable paragraph or two. For the sake of brevity, this section should omit discussion of conﬁguration details in favor of providing a more 10,000 foot view of the plugin features.
Purpose
Why would someone want to use this particular plugin? What problems does it solve, or what features not attainable otherwise (through core functionality or other plugins) does it provide to an administrator or user? While not exactly a product pitch, this is the section in which the case for using the plugin at all should be made.
486 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
Dependencies
If the plugin has external dependencies, they should be listed in full, with the exception of items already required by a bare-bones Trafﬁc Server build. If the plugin has no unusual or non-standard dependencies, this section may be safely omitted. For example, there is no need to list a C++ compiler as a requirement, but if the plugin requires OpenCL and a speciﬁc brand of GPU expansion card to perform some wacky image transformations on cached objects - that really should be disclosed. If any of the dependencies are optional, that should be noted as well as the functionality that may be missing should the dependency not be present.
Installation
Detailed instructions on how to install the plugin if it is not built by default with Trafﬁc Server or requires any additional libraries, services, or applications outside of Trafﬁc Server to be present. If the plugin is marked stable and included by default in Trafﬁc Server builds, then it may not be necessary to include any installation steps beyond noting that it is part of the default Trafﬁc Server.
Conﬁguration
Detailed instructions on how and where to provide conﬁguration data for the plugin’s features. It is important to use reStructuredText references for all mentions of Trafﬁc Server conﬁguration variables, as well as their containing conﬁguration ﬁles. If conﬁgurations are generated by scripts included with the plugin source code, those scripts should be mentioned and their usage should be shown.
Caveats (or Limitations)
Notes about any peculiarities with the plugin, incompatibilities or strange behaviors when used in conjunction with other features, or special considerations for its deployment that might affect an administrator’s use of the plugin or their infrastructure’s architecture. If the plugin contains known limitations (perhaps it only works with some protocols, can understand only some media types, or is especially non-performant under speciﬁc conﬁgurations) they should be noted. If possible, workarounds for these limitations should be suggested, or if none are available that should be noted.
Examples
Every plugin should aim to have at least one, complete example of its usage beginning with its basic installation, showing a complete (and ideally working) conﬁguration, as well as sample output from Trafﬁc Server in which the plugin’s effects may be observed.
Further Reading
Consolidated list of references to outside materials and documents which may be relevant to the plugin, such as RFCs and white papers. If the plugin is related to external projects, or has been the subject of benchmarks or articles, those may be useful to list as well.
4.11. Documentation 487 Apache-Trafﬁcserver-Server Documentation, latest
Additional Sections
Not all plugins will ﬁt perfectly within that list of sections. Some may require additional details, or such extended discussion that it is best to break the plugin’s documentation down a bit further. In general, it is recommended to place these additional documentation sections in between Conﬁguration and Caveats or Limitations. This ordering permits a quick scanning of the plugin’s purpose, basic method of installation, and initial overview of its conﬁguration before becoming mired down in dissertation length details of every possible conﬁguration, warning, and trade-off that may affect deployments. In cases where the array of conﬁguration possibilities is extensive, it is recommended that the initial Conﬁguration section describe the most common conﬁguration scenario(s), while noting that there are many other possibilities that will be covered in the sections that follow. This, it is hoped, will get the usual case out of the way quickly for most Trafﬁc Server administrators while limiting the paralysis of choice they may face otherwise.
Documentation Template
For the convenience of plugin developers, this manual includes a plugin documentation template which you may use as a base. The template includes placeholders, with markup, for common features of the documentation which may be relevant for your plugin. Developers are very strongly encouraged to use this template whenever possible, as a means of producing and maintaining a consistent format for plugin documentation.
Host Resolution Proposal
Introduction
The current mechanism for resolving host names to IP addresses for Trafﬁc Server is contained the HostDB and DNS libraries. These take hostnames and provide IP addresses for them. The current implementation is generally considered inadequate, both from a functionality point of view and difﬁculty in working with it in other parts of Trafﬁc Server. As Trafﬁc Server is used in more complex situtations this inadequacy presents increasing problems.
Goals
Updating the host name resolution (currently referred to as “HostDB”) has several functions goals • Enable additional processing layers to be easily added. • Enable plugins to directly access the name resolution logic • Enable plugins to provide name resolution • Asynchronous (immediate resolve or callback on block) • Minimize allocations – in particular no allocations for cached resolutions • Simplify interactions with the resolution, particularly with regard to nameservers, origin server failover, and address family handling. It is also necessary to support a number of speciﬁc features that are either currently available or strongly desired. • SplitDNS or its equivalent • Use of a hosts ﬁle (e.g. /etc/hosts) • Simultaneous IPv4 and IPv6 queries
488 Chapter 4. Developer’s Guide Apache-Trafﬁcserver-Server Documentation, latest
• IP family control • Negative caching * Server connection failures * Query failures * Nameserver failures. • Address validity time out control • Address round robin support • SRV record support (weighted records) • Nameserver round robin • Plugin access to nameserver data (add, remove, enumerate) • Plugin provision of resolvers. • Hooks for plugin detection / recovery from resolution events. One issue is persistence of the cached resolutions. This creates problems for the current implementation (because of size limits it imposes on the cached data) but also allows for quicker restarts in a busy environment.
Basics
The basic design is to separate the functionality into chainable layers so that a resolver with the desired attributes can be assembled from those layers. The core interface is that of a lazy iterator. This object returns one of four results when asked for an address • An IP address • Done(no more addresses are available) • Wait(an address may be available in the future) • Fail (no address is available and none will be so in the future) Each layer (except the bottom) uses this API and also provides it. This enables higher level logic such as the state machine to simply use the resolver as a list without having to backtrack states in the case of failures, or have special cases for different resolution sources. To perform a resolution, a client creates a query object (potentially on the stack), initializes it with the required data (at least the hostname) and then starts the resolution. Methods on the query object allow its state and IP address data to be accessed.
Required Resolvers
Nameserver A bottom level resolver that directly queries a nameserver for DNS data. This contains much of the functionality currently in the iocore/dns directory. SplitDNS A resolver that directs requests to one of several resolvers. To emulate current behavior these would be Nameserver instances. NameserverGroup A grouping mechanism for Nameserver instances that provides failover, round robin, and order- ing capabilities. It may be reasonable to merge this with the SplitDNS resolver. HostFile A resolver that uses a local ﬁle to resolve names. AddressCache A resolver that also has a cache for resolution results. It requires another resolver instance to perform the actual resolution. Preloaded A resolver that can contain one or more explicitly set IP addresses which are returned. When those are exhausted it falls back to another resolver.
4.12. Host Resolution Proposal 489 Apache-Trafﬁcserver-Server Documentation, latest
Conﬁguration
To conﬁguration the resolution, each resolver would be assigned a tag. It is not, however, sufﬁcient to simply provide the list of resolver tags because some resolvers require additional conﬁguration. Unfortunately this will likely require a separate conﬁguration ﬁle outside of records.config, although we would be able to remove splitdns. config. In this case we would need chain start / end markers around a list of resolver tags. Each tag would the be able to take additional resolver conﬁguration data. For instance, for a SplitDNS resolver the nameservers.
Examples
Transparent operations would beneﬁt from the Preloaded resolver. This would be loaded with the origin host address provided by the client connection. This could be done early in processing and then no more logic would be required to skip DNS processing as it would happen without additional action by the state machine. It would handle the problem of de facto denial of service if an origin server becomes unavailable in that conﬁguration, as Preloaded would switch to alternate addresses automatically. Adding host ﬁle access would be easier as well, as it could be done in a much more modular fashion and then added to the stack at conﬁguration time. Whether such addresses were cached would be controlled by chain arrangement rather yet more conﬁguration knobs. The default conﬁguration would be Preloaded : AddressCache : Nameserver. In all cases the state machine makes requests against the request object to get IP addresses as needed.
Issues
Request object allocation
The biggest hurdle is being able to unwind a resolver chain when a block is encountered. There are some ways to deal with this. 1) Set a maximum resolver chain length and declare the request instance so that there is storage for state for that many resolvers. If needed and additional value of maximum storage per chain could be set as well. The expected number of elements in a chain is expected to be limited, 10 would likely be a reaosnable limit. If settable at source conﬁguration time this should be sufﬁcient. 2) Embed class allocators in resolver chains and mark the top / outermost / ﬁrst resolver. The maximum state size for a resolution can be calculated when the chain is created and then the top level resolver can use an allocation pool to efﬁciently allocate request objects. This has an advantage that with a wrapper class the request object can be passed along cheaply. Whether that’s an advantage in practice is unclear.
Plugin resolvers
If plugins can provide resolvers, how can these can integrated in to existing resolver chains for use by the HTTP SM for instance?
Feedback
It should be possible for a client to provide feedback about addresses (e.g., the origin server at this address is not available). Not all resolvers will handle feedback but some will and that must be possible. Related to this is that caching resolvers (such as AddressCache) must be able to iterator over all resolved addresses even if their client does not ask for them. In effect they must background ﬁll the address data.
490 Chapter 4. Developer’s Guide CHAPTER 5
Appendices
Command Line Utilities
 trafﬁc_cop
Description
 traffic_cop is a watchdog program that is responsible for starting trafﬁc_manager and trafﬁc_server and moni- toring them for responsiveness. If either of these processes are determined to be unresponsive, traffic_cop will kill and restart them. On Linux, traffic_cop will also monitor available memory and swap space, restarting the watched pro- cesses if the available memory falls below a minimum threshold. The memory thresholds can be conﬁgured with the :ts:cv:‘proxy.conﬁg.cop.linux_min_swapfree_kb‘ and :ts:cv:‘proxy.conﬁg.cop.linux_min_memfree_kb‘ vari- ables.
Options
-d, --debug Emit debugging messages. -o, --stdout traffic_cop ordinarily logs to syslog, however for debugging purposes, this option causes it to print mes- sages to standard output instead. -s, --stop Kill children using SIGSTOP instead of SIGKILL. This option is primarily for debugging. -V, --version Print version information and exit.
491 Apache-Trafﬁcserver-Server Documentation, latest
See also
 syslog(1), traffic_manager(8), traffic_server(8)
 trafﬁc_crashlog
Synopsis
 traffic_crashlog [options]
Description
 traffic_crashlog is a helper process that catches Trafﬁc Server crashes and writes a crash report log to the logging directory. Other than for debugging or development purposes, traffic_crashlog is not intended for users to run directly. When trafﬁc_server starts, it will launch a traffic_crashlog process and keep it stopped, activating it only if a crash occurs.
Options
--host TRIPLE This option speciﬁes the host triple for the process that traffic_crashlog should examine. If a supported host triple is speciﬁed, traffic_crashlog expects to receive a siginfo_t structure on it’s standard input, followed by a ucontext_t. --target PID Speciﬁes the process ID of the crashing traffic_server process. If this option is not speciﬁed, traffic_crashlog assumes it should examine it’s parent process. --syslog This option causes traffic_crashlog to log the name of the crash log it creates to the system log. --debug This option enables debugging mode. In this mode, traffic_crashlog emits the log to it’s standard output. --wait This option causes traffic_crashlog to stop itself immediately after it is launched. traffic_server will allow it to continue only when a crash occurs.
Caveats
 traffic_crashlog makes use of various Trafﬁc Server management APIs. If trafﬁc_manager is not available, the crash log will be incomplete. traffic_crashlog may generate a crash log containing information you would rather not share outside your organization. Please examine the crash log carefully before posting it in a public forum.
See also
 records.config(5), traffic_manager(8), traffic_server(8)
492 Chapter 5. Appendices Apache-Trafﬁcserver-Server Documentation, latest
 trafﬁc_ctl
Synopsis
 traffic_ctl [OPTIONS] SUBCOMMAND [OPTIONS]
Description
 traffic_ctl is used to display and manipulate conﬁgure a running Trafﬁc Server. traffic_ctl includes a number of subcommands that control different aspects of Trafﬁc Server: traffic_ctl alarm Display and manipulate Trafﬁc Server alarms traffic_ctl cluster Stop, restart and examine the cluster traffic_ctl config Manipulate and display conﬁguration records traffic_ctl metric Manipulate performance and status metrics traffic_ctl server Stop, restart and examine the server traffic_ctl storage Manipulate cache storage
Options
--debug Enable debugging output. -V, --version Print version information and exit.
Subcommands trafﬁc_ctl alarm list List all alarm events that have not been acknowledged (cleared). clear Clear (acknowledge) all current alarms. resolve ALARM [ALARM...] Clear (acknowledge) an alarm event. The arguments are a speciﬁc alarm number (e.g. “1’‘), or an alarm string identiﬁer (e.g. ‘’MGMT_ALARM_PROXY_CONFIG_ERROR’‘). trafﬁc_ctl cluster restart [--drain] [--manager] Shut down and immediately restart Trafﬁc Server, node by node across the cluster. The –drain and –manager options have the same behavior as for the traffic_ctl server restart subcommand. status Show the current cluster status.
5.1. Command Line Utilities 493 Apache-Trafﬁcserver-Server Documentation, latest
 trafﬁc_ctl conﬁg
 defaults [--records] Display the default values for all conﬁguration records. The –records ﬂag has the same behavior as traffic_ctl config get --records. describe RECORD [RECORD...] Display all the known information about a conﬁguration record. This includes the current and default values, the data type, the record class and syntax checking expression. diff [--records] Display conﬁguration records that have non-default values. The –records ﬂag has the same behavior as traffic_ctl config get --records. get [--records] RECORD [RECORD...] Display the current value of a conﬁguration record. --records If this ﬂag is provided, traffic_ctl config get will emit results in records.config format. match [--records] REGEX [REGEX...] Display the current values of all conﬁguration variables whose names match the given regular expression. The –records ﬂag has the same behavior as traffic_ctl config get --records. reload Initiate a Trafﬁc Server conﬁguration reload. Use this command to update the running conﬁguration after any conﬁguration ﬁle modiﬁcation. The timestamp of the last reconﬁguration event (in seconds since epoch) is published in the proxy.node.conﬁg.reconﬁgure_time metric. set RECORD VALUE Set the named conﬁguration record to the speciﬁed value. Refer to the records.config documentation for a list of the conﬁguration variables you can specify. Note that this is not a synchronous operation. status Display detailed status about the Trafﬁc Server conﬁguration system. This includes version information, whether the internal conﬁguration store is current and whether any daemon processes should be restarted.
 trafﬁc_ctl metric
 clear [--cluster] Reset all statistics to zero. The –cluster option applies this across all cluster nodes. get METRIC [METRIC...] Display the current value of the speciﬁes statistics. match REGEX [REGEX...] Display the current values of all statistics whose names match the given regular expression. zero [--cluster] METRIC [METRIC...] Reset the named statistics to zero. The –cluster option applies this across all cluster nodes.
 trafﬁc_ctl server
 restart Shut down and immediately restart Trafﬁc Server
494 Chapter 5. Appendices Apache-Trafﬁcserver-Server Documentation, latest
--drain This option modiﬁes the behavior of traffic_ctl server restart such that traffic_server is not shut down until the number of active client connections drops to the number given by the :ts:cv:‘proxy.conﬁg.restart.active_client_threshold‘ conﬁguration variable. --manager The default behavior of traffic_ctl server restart is to restart traffic_server. If this option is speciﬁed, traffic_manager is also restarted. status Show the current proxy server status, indicating if we’re running or not. backtrace Show a full stack trace of all the traffic_server threads.
 trafﬁc_ctl storage
 offline DEVICE [DEVICE ...] Mark a cache storage device as ofﬂine. The storage is identiﬁed by a path which must match exactly a path speciﬁed in storage.config. This removes the storage from the cache and redirects requests that would have used this storage to other storage. This has exactly the same effect as a disk failure for that storage. This does not persist across restarts of the traffic_server process.
Examples
Conﬁgure Trafﬁc Server to log in Squid format:
$ traffic_ctl config set proxy.config.log.squid_log_enabled 1 $ traffic_ctl config set proxy.config.log.squid_log_is_ascii 1 $ traffic_ctl config reload
See also
 records.config(5), storage.config(5)
 trafﬁc_line
: This utility is deprecated as of v6.0.0, and replaced with traffic_ctl. You should change your tools / scripts to use this new application instead.
Synopsis
 traffic_line [options]
Description
 traffic_line is used to execute individual Trafﬁc Server commands and to script multiple commands in a shell.
5.1. Command Line Utilities 495 Apache-Trafﬁcserver-Server Documentation, latest
Options
-B, --bounce_cluster Bounce all Trafﬁc Server nodes in the cluster. Bouncing Trafﬁc Server shuts down and immediately restarts Trafﬁc Server, node-by-node. -b, --bounce_local Bounce Trafﬁc Server on the local node. Bouncing Trafﬁc Server shuts down and immediately restarts the Trafﬁc Server node. -C, --clear_cluster Clears accumulated statistics on all nodes in the cluster. -c, --clear_node Clears accumulated statistics on the local node. --drain This option modiﬁes the behavior of traffic_line -b and traffic_line -L such that traffic_server is not shut down until the number of active client connections drops to the number given by the :ts:cv:‘proxy.conﬁg.restart.active_client_threshold‘ conﬁguration variable. -h, --help Print usage information and exit. -L, --restart_local Restart the traffic_manager and traffic_server processes on the local node. -M, --restart_cluster Restart the traffic_manager process and the traffic_server process on all the nodes in a cluster. -m REGEX, --match_var REGEX Display the current values of all performance statistics or conﬁguration variables whose names match the given regular expression. -r VAR, --read_var VAR Display speciﬁc performance statistics or a current conﬁguration setting. -s VAR, --set_var VAR Set the conﬁguration variable named VAR. The value of the conﬁguration variable is given by the traffic_line -v option. Refer to the records.config documentation for a list of the conﬁguration variables you can specify. -S, --shutdown Shut down Trafﬁc Server on the local node. -U, --startup Start Trafﬁc Server on the local node. -v VALUE, --value VALUE Specify the value to set when setting a conﬁguration variable. -V, --version Print version information and exit. -x, --reread_config Initiate a Trafﬁc Server conﬁguration ﬁle reread. Use this command to update the running conﬁguration after any conﬁguration ﬁle modiﬁcation. The timestamp of the last reconﬁguration event (in seconds since epoch) is published in the proxy.node.conﬁg.reconﬁgure_time metric.
496 Chapter 5. Appendices Apache-Trafﬁcserver-Server Documentation, latest
-Z, --zero_cluster Reset performance statistics to zero across the cluster. -z, --zero_node Reset performance statistics to zero on the local node. --offline PATH Mark a cache storage device as ofﬂine. The storage is identiﬁed by a path which must match exactly a path speciﬁed in storage.config. This removes the storage from the cache and redirects requests that would have used this storage to other storage. This has exactly the same effect as a disk failure for that storage. This does not persist across restarts of the traffic_server process. --alarms List all alarm events that have not been acknowledged (cleared). --clear_alarms [all | #event | name] Clear (acknowledge) an alarm event. The arguments are “all” for all current alarms, a speciﬁc alarm number (e.g. “1’‘), or an alarm string identiﬁer (e.g. ‘’MGMT_ALARM_PROXY_CONFIG_ERROR’‘). --status Show the current proxy server status, indicating if we’re running or not.
Performance Statistics proxy.process.ssl.user_agent_other_errors Total number of other ssl client connection errors (counts ssl errors that are not captured in other user agent stats below) proxy.process.ssl.user_agent_expired_cert Total number of ssl client connection failures where the cert was ex- pired. proxy.process.ssl.user_agent_revoked_cert Total number of ssl client connection failures where the cert was re- voked. proxy.process.ssl.user_agent_unknown_cert Total number of ssl client connection failures related to the cert, but speciﬁc error was unknown. proxy.process.ssl.user_agent_cert_verify_failed Total number of ssl client connection failures where cert veriﬁca- tion failed. proxy.process.ssl.user_agent_bad_cert Total number of ssl client connection failures where the cert is bad. proxy.process.ssl.user_agent_decryption_failed Total number of ssl client connection decryption failures (during negotiation). proxy.process.ssl.user_agent_wrong_version Total number of ssl client connections that provided an invalid proto- col version. proxy.process.ssl.user_agent_unknown_ca Total number of ssl client connection that failed due to unknown ca. proxy.process.ssl.origin_server_other_errors Total number of other ssl origin server connection errors (counts ssl errors that are not captured in other origin server stats below). proxy.process.ssl.origin_server_expired_cert Total number of ssl origin server connection failures where the cert was expired. proxy.process.ssl.origin_server_revoked_cert Total number of ssl origin server connection failures where the cert was revoked. proxy.process.ssl.origin_server_unknown_cert Total number of ssl origin server connection failures related to the cert where speciﬁc error was unknown.
5.1. Command Line Utilities 497 Apache-Trafﬁcserver-Server Documentation, latest
 proxy.process.ssl.origin_server_cert_verify_failed Total number of ssl origin server connection failures where cert veriﬁcation failed. proxy.process.ssl.origin_server_bad_cert Total number of ssl origin server connection failures where the cert is bad. proxy.process.ssl.origin_server_decryption_failed Total number of ssl origin server connection decryption failures (during negotiation). proxy.process.ssl.origin_server_wrong_version Total number of ssl origin server connections that provided an in- valid protocol version. proxy.process.ssl.origin_server_unknown_ca Total number of ssl origin server connection that failed due to un- known ca. proxy.process.ssl.user_agent_sessions Total number of ssl/tls sessions created. proxy.process.ssl.user_agent_session_hit Total number of session hits. A previous session was reused which re- sulted in an abbreviated ssl client negotiation. proxy.process.ssl.user_agent_session_miss Total number of session misses. The ssl client provided a session id that was not found in cache and, therefore, could not be used. proxy.process.ssl.user_agent_session_timeout Total number of session timeouts. The ssl client provided a session, but it could not be used because it was past the session timeout. proxy.process.ssl.cipher.user_agent.{CIPHERNAME} Total number of ssl client connections that used ci- pherName. The list of cipher statistics is dynamic and depends upon the installed ciphers and the :ts:cv:‘proxy.conﬁg.ssl.server.cipher_suite‘ conﬁguration. The set of cipher statistics can be discovered with traffic_line -m. For example:
$ traffic_line -m proxy.process.ssl.cipher.user_agent. proxy.process.ssl.cipher.user_agent.ECDHE-RSA-AES256-GCM-SHA384 0 proxy.process.ssl.cipher.user_agent.ECDHE-ECDSA-AES256-GCM-SHA384 0 proxy.process.ssl.cipher.user_agent.ECDHE-RSA-AES256-SHA384 0 proxy.process.ssl.cipher.user_agent.ECDHE-ECDSA-AES256-SHA384 0 ...
Cache Statistics
Cache statistics come in two varieties, global and per cache volume. These will be listed here in the global form. To get a cache volume statistic add .volume_# to the name after cache where # is 1-based index of the volume in storage.config. For example the statistic proxy.process.cache.sync.bytes is a global statistic. The value for the third cache volume is proxy.process.cache.volume_3.sync.bytes. proxy.process.cache.sync.bytes The total number of bytes written to disk to synchronize the cache directory. proxy.process.cache.sync.time The total time, in nanoseconds, during which the cache directory was being written to disk. proxy.process.cache.sync.count The number of times a cache directory sync has been done. proxy.process.cache.wrap_count The number of times a cache stripe has cycled. Each stripe is a circular buffer and this is incremented each time the write cursor is reset to the start of the stripe.
Examples
Conﬁgure Trafﬁc Server to log in Squid format:
498 Chapter 5. Appendices Apache-Trafﬁcserver-Server Documentation, latest
$ traffic_line -s proxy.config.log.squid_log_enabled -v 1 $ traffic_line -s proxy.config.log.squid_log_is_ascii -v 1 $ traffic_ctl config reload
See also
 records.config(5), storage.config(5)
 trafﬁc_logcat
Synopsis
 traffic_logcat [-o output-ﬁle | -a] [-CEhSVw2] [input-ﬁle ...]
Description
To analyse a binary log ﬁle using standard tools, you must ﬁrst convert it to ASCII. traffic_logcat does exactly that.
Options
-o PATH, --output_file PATH Speciﬁes where the command output is directed. -a, --auto_filename Automatically generates the output ﬁlename based on the input ﬁlename. If the input is from stdin, then this option is ignored. For example:
 traffic_logcat-a squid-1.blog squid-2.blog squid-3.blog
 generates:
 squid-1.log squid-2.log squid-3.log
-f, --follow Follows the ﬁle, like tail(1) -f -C, --clf Attempts to transform the input to Netscape Common format, if possible. -E, --elf Attempts to transform the input to Netscape Extended format, if possible. -S, --squid Attempts to transform the input to Squid format, if possible. -2, --elf2 Attempt to transform the input to Netscape Extended-2 format, if possible. -T, --debug_tags
5.1. Command Line Utilities 499 Apache-Trafﬁcserver-Server Documentation, latest
-w, --overwrite_output -h, --help Print usage information and exit. -V, --version Print version information and exit.
: Use only one of the following options at any given time: -S, -C, -E, or -2.
If no input ﬁles are speciﬁed, then traffic_logcat reads from the standard input (stdin). If you do not specify an output ﬁle, then traffic_logcat writes to the standard output (stdout). For example, to convert a binary log ﬁle to an ASCII ﬁle, you can use the traffic_logcat command with either of the following options below:
 traffic_logcat binary_file> ascii_file traffic_logcat-o ascii_file binary_file
The binary log ﬁle is not modiﬁed by this command.
See Also
 tail(1)
 trafﬁc_logstats
Description
Options
-f FILE, --log_file FILE -o LIST, --origin_list LIST -O FILE, --origin_file FILE -M COUNT, --max_origins COUNT -u COUNT, --urls COUNT -U COUNT, --show_urls COUNT --A, --as_object -i, --incremental -S FILE, --statetag FILE -t, --tail -s, --summary -j, --json -c, --cgi -m, --min_hits -a, --max_age
500 Chapter 5. Appendices Apache-Trafﬁcserver-Server Documentation, latest
-l COUNT, --line_len COUNT -T TAGS, --debug_tags TAGS -h, --help Print usage information and exit. -V, --version Print version information and exit. trafﬁc_manager
Description
--aconfPort PORT --action TAGS --clusterMCPort PORT --clusterRSPort PORT --debug TAGS --groupAddr ADDRESS --help --nosyslog --path FILE --proxyBackDoor PORT --proxyOff --proxyPort PORT --recordsConf FILE --tsArgs ARGUMENTS --version
Environment
MGMT_ACONF_PORT MGMT_CLUSTER_MC_PORT MGMT_CLUSTER_RS_PORT MGMT_GROUP_ADDR
Signals
SIGHUP This signal causes a reconﬁguration event, equivalent to running traffic_ctl config reload.
See also traffic_ctl(8)
5.1. Command Line Utilities 501 Apache-Trafﬁcserver-Server Documentation, latest trafﬁc_server
Description
Options
-n COUNT, --net_threads COUNT -Z COUNT, --cluster_threads COUNT -U COUNT, --udp_threads COUNT -a, --accepts_thread -b, --accept_till_done -p PORT, --httpport PORT -P PORT, --cluster_port PORT -f, --disable_freelist In order to improve performance, traffic_server caches commonly used data structures in a set of free object lists. This option disables these caches, causing traffic_server to use malloc(3) for every allocation. Though this option should not commonly be needed, it may be beneﬁcial in memory-constrained environments or where the working set is highly variable. -o LEVEL, --dprintf_level LEVEL -R LEVEL, --regression LEVEL -r TEST, --regression_rest TEST -T TAGS, --debug_tags TAGS -B TAGS, --action_tags TAGS -i COUNT, --interval COUNT -M, --remote_management -C CMD, --command CMD -k, --clear_hostdb -K, --clear_cache -c CORE, --read_core CORE --accept_mss MSS -t MSECS, --poll_timeout MSECS -h, --help Print usage information and exit. -V, --version Print version information and exit.
Environment
PROXY_REMOTE_MGMT PROXY_AUTO_EXIT
502 Chapter 5. Appendices Apache-Trafﬁcserver-Server Documentation, latest trafﬁc_top
Description
Options
-s COUNT trafﬁc_via
Synopsis traffic_via [OPTIONS] [VIA ...]
Description traffic_via decodes the Trafﬁc Server Via header codes. Via header strings are passed as command-line options. The enclosing square brackets are optional. If the argument is ‘-‘, traffic_via will ﬁlter standard input for Via headers. This modes supports only supports Via headers that are enclosed by square brackets.
Options
-h, --help Print usage information and exit. -V, --version Print version.
Examples
Decode the Via header from command-line arguments:
$ traffic_via "[uScMsEf p eC:t cCMi p sF]" Via header is uScMsEf p eC:t cCMi p sF, Length is 24 Via Header Details: Request headers received from client :simple request (not
˓→conditional) Result of Traffic Server cache lookup for URL :miss (a cache "MISS") Response information received from origin server :error in response Result of document write-to-cache: :no cache write performed Proxy operation result :unknown Error codes (if any) :connection to server failed Tunnel info :no tunneling Cache Type :cache Cache Lookup Result :cache miss (url not in cache) ICP status :no icp Parent proxy connection status :no parent proxy or unknown Origin server connection status :connection open failed
Decode the Via header from a curl request, using the X-Debug plugin:
5.1. Command Line Utilities 503 Apache-Trafﬁcserver-Server Documentation, latest
$ curl -H "X-Debug: Via" -I http://test.example.com | traffic_via - Via header is uScMsSf pSeN:t cCMi p sS, Length is 24 Via Header Details: Request headers received from client :simple request (not
˓→conditional) Result of Traffic Server cache lookup for URL :miss (a cache "MISS") Response information received from origin server :connection opened successfully Result of document write-to-cache: :no cache write performed Proxy operation result :served or connection opened
˓→successfully Error codes (if any) :no error Tunnel info :no tunneling Cache Type :cache Cache Lookup Result :cache miss (url not in cache) ICP status :no icp Parent proxy connection status :no parent proxy or unknown Origin server connection status :connection opened successfully trafﬁc_wccp
Description
Front end to the wccp client library. It is a stand alone program that speaks the client side of the WCCP cache protocol. It can be used instead of the built in WCCP feature in Trafﬁc Server. This can be beneﬁcial if you have multiple programs running on the same computer that are relying on WCCP to redirect trafﬁc from the router to the computer. Since it relies on the wccp library, traffic_wccp is only built if Trafﬁc Server is conﬁgured with --enable-wccp. The overall Apache Trafﬁc Server WCCP conﬁguration documentation is at WCCP Conﬁguration
Options
--address IP address to bind. --router Booststrap IP address for routers. --service Path to service group definitions. --debug Print debugging information. --daemon Run as daemon. You need to run at least with the --service arguments. An example service deﬁnition ﬁle, service-nogre-example.config, is included in the cmd/traffic_wccp directory. In this ﬁle you de- ﬁne your MD5 security password (highly recommended), and you deﬁne your service groups. The details of the service ﬁle are deﬁned at WCCP Service Conﬁguration.
Limitations
The current WCCP implementation associated with ATS only supports one cache client per service group per router. The cache assignment logic currently assigns the current cache client to all buckets.
504 Chapter 5. Appendices Apache-Trafﬁcserver-Server Documentation, latest
 tspush
Description
Options
-f FILE, --file FILE -u URL, --url URL -v, --verbose -h, --help Print usage information and exit.
 tsxs
Description
Options
-q VAR, --query VAR -I PATH -L PATH -l LIB -o FILE -i -c FILE [FILE ...] -C FILE [FILE ...] -v, --verbose -h, --help Print usage information and exit.
Frequently Asked Questions
How do you create a raw disk for the cache if all your disks have mounted ﬁle systems?
Create a large ﬁle on ﬁlesystem (with dd(1)) and mount it as loopback device. This is accomplished with losetup(8) on Linux, lofiadm(1m) on Solaris and Illumos, and mdconfig(8) on FreeBSD.
How do disk I/O errors affect the cache and what does Trafﬁc Server do when a cache disk fails?
If a disk drive fails ﬁve successive I/O operations, then Trafﬁc Server considers the drive inaccessible and removes the entire disk from the cache. Normal cache operations continue for all other Trafﬁc Server disk drives.
5.2. Frequently Asked Questions 505 Apache-Trafﬁcserver-Server Documentation, latest
If a client disconnects during the time that Trafﬁc Server is downloading a large object, is any of the object saved in the cache?
When a client disconnects during an HTTP operation, Trafﬁc Server can continue to down- load the object from the origin server, using the background ﬁll feature. It will con- tinue downloading based on the :ts:cv:‘proxy.conﬁg.http.background_ﬁll_active_timeout‘ and :ts:cv:‘proxy.conﬁg.http.background_ﬁll_completed_threshold‘ settings.
Can Trafﬁc Server cache Java applets, JavaScript programs, or other application ﬁles like VBScript?
Yes, Trafﬁc Server can store and serve Java applets, JavaScript programs, VBScripts, and other executable objects from its cache according to the freshness and cacheability rules for HTTP objects. Trafﬁc Server does not execute the applets, scripts, or programs. These objects run entirely client-side (the system which originated the request for the objects), and do not execute on the server.
In Squid- and Netscape-format log ﬁles, what do the cache result codes mean?
This is described in detail in the Squid documentation.
What is recorded by the cqtx ﬁeld in a custom log ﬁle?
• In forward proxy mode, the cqtx ﬁeld records the complete client request in the log ﬁle (for example, GET http://www.company.com HTTP/1.0). • In reverse proxy mode, the cqtx ﬁeld records the hostname or IP address of the origin server because Trafﬁc Server ﬁrst remaps the request as per map rules in the remap.config ﬁle.
Does Trafﬁc Server refresh entries in its host database after a certain period of time if they have not been used?
By default, the Trafﬁc Server host database observes the time-to-live (ttl) values set by name servers. You can reconﬁgure Trafﬁc Server to ignore the ttl set by name servers and use a speciﬁc Trafﬁc Server setting instead. Alternatively, you can conﬁgure Trafﬁc Server to compare the ttl value set by the name server with the ttl value set by Trafﬁc Server, and then use either the lower or the higher value. Refer to :ts:cv:‘proxy.conﬁg.hostdb.ttl_mode‘ for more info.
Can you improve the look of your custom response pages by using images, ani- mated .gifs, and Java applets?
No, because Trafﬁc Server can only respond to clients with a single text or HTML document. As a workaround, however, you can provide references on your custom response pages to images, animated .gifs, Java applets, or objects other than text which are located on a web server. Add links in the body_factory template ﬁles in the same way you would for any image in an HTML document (i.e., with the full URL in the SRC attribute).
506 Chapter 5. Appendices Apache-Trafﬁcserver-Server Documentation, latest
Can Trafﬁc Server run in forward proxy and reverse proxy modes at the same time?
Yes. When you enable reverse proxy mode, Trafﬁc Server remaps incoming requests according to the map rules in remap.config. All other requests that do not match a map rule are served in forward proxy mode. If you want to run in reverse proxy only mode (wherein Trafﬁc Server does not serve requests that fail to match a map rule), then you must set the conﬁguration variable :ts:cv:‘proxy.conﬁg.url_remap.remap_required‘ to 1 in records.config.
How do I enable forward proxy mode
Please refer to the Forward Proxy documentation.
How do I interpret the Via: header code?
The Via header string can be decoded with the Via Decoder Ring. The Via header is an optional HTTP header added by Trafﬁc Server and other HTTP proxies. If a request goes through multiple proxies, each one appends its Via header content to the end of the existing Via header. Via header content is for general information and diagnostic use only and should not be used as a programmatic interface to Trafﬁc Server. The form of the Via header is Via:   ( []) Value Meaning  the scheme and version of the HTTP request  the conﬁgured name of the proxy server   a string of alphabetic codes presenting status information about the proxy handling of the HTTP request For example: Via: HTTP/1.0 storm (Trafﬁc-Server/4.0.0 [cMs f ]) • [u lH o f pS eN] cache hit • [u lM oS fF pS eN] cache miss • [uN l oS f pS eN] no-cache origin server fetch The short status code shows the cache-lookup, server-info and cache-ﬁll information as listed in the full status code description below. The long status code list provided in older, commercial versions of Trafﬁc Server can be restored by setting the verbose_via_str conﬁg variable. The character strings in the via-code show [:] where  represents status information about the results of the client request and  represent some information about the proxy operations performed during request processing. The full via-code status format is [u c s f p e : tc i p s] u client-info
Request headers received from client. Value is one of:
5.2. Frequently Asked Questions 507 Apache-Trafﬁcserver-Server Documentation, latest
Value Meaning C cookie E error in request I If Modiﬁed Since (IMS) N no-cache S simple request (not conditional) c cache-lookup
Result of Trafﬁc Server cache lookup for URL. Value is one of: Value Meaning A in cache, not acceptable (a cache “MISS”) H in cache, fresh (a cache “HIT”) M miss (a cache “MISS”) R in cache, fresh RAM hit (a cache “HIT”) S in cache, stale (a cache “MISS”) blank no cache lookup performed s server-info
Response information received from origin server. Value is one of: Value Meaning E error in response N not-modiﬁed S served blank no server connection needed f cache-ﬁll
Result of document write to cache. Value is one of: Value Meaning D cached copy deleted U updated old cache copy W written into cache (new copy) blank no cache write performed p proxy-info
Proxy operation result. Value is one of: Value Meaning N not-modiﬁed R origin server revalidated S served e error-codes
Value is one of:
508 Chapter 5. Appendices Apache-Trafﬁcserver-Server Documentation, latest
Value Meaning A authorization failure C connection to server failed D dns failure F request forbidden H header syntax unacceptable N no error R cache read error M moved temporarily S server related error T connection timed out : = Separates proxy request result information from operation detail codes t tunnel-info
Proxy-only service operation. Value is one of: Value Meaning A tunnel authorization F tunneling due to a header ﬁeld (such as presence of If-Range header) M tunneling due to a method (e.g. CONNECT) N tunneling due to no forward O tunneling because cache is turned off U tunneling because of url (url suggests dynamic content) blank no tunneling c cache-type and cache-lookup cache result values (2 characters) cache-type character value is one of Value Meaning C cache I icp L cluster, (not used) P parent S server blank cache miss or no cache lookup cache-lookup-result character value is one of: Value Meaning C cache hit, but conﬁg forces revalidate D cache hit, but method forces revalidated (e.g. ftp, not anonymous) H cache hit I conditional miss (client sent conditional, fresh in cache, returned 412) K cookie miss M cache miss (url not in cache) N conditional hit (client sent conditional, doc fresh in cache, returned 304) S cache hit, but expired U cache hit, but client forces revalidate (e.g. Pragma: no-cache) blank no cache lookup
5.2. Frequently Asked Questions 509 Apache-Trafﬁcserver-Server Documentation, latest i icp-conn-info
ICP status Value Meaning F connection open failed S connection opened successfully blank no icp p parent-proxy parent proxy connection status Value Meaning F connection open failed S connection opened successfully blank no parent proxy s server-conn-info origin server connection status Value Meaning F connection open failed S connection opened successfully blank no server connection
Support for HTTP Expect: Header
Trafﬁc Server currently does not handle Expect: request headers according to the HTTP/1.1 spec. Clients such as cURL automatically send Expect: for POST requests with large POST bodies, with a 1 second timeout if a 100 Continue response is not received. To avoid the timeout when using cURL as a client to Trafﬁc Server, you can turn off the Expect: header: curl-H"Expect:" http://www.example.com/
Or with the C (libcurl) library from within your own applications: struct curl_slist *header_list=NULL; header_list= curl_slist_append(header_list,"Expect:"); curl_easy_setopt(my_curlp, CURLOPT_HTTPHEADER, header_list);
Or with the PHP cURL library: curl_setopt($ch, CURLOPT_HTTPHEADER, array('Expect:'));
510 Chapter 5. Appendices Apache-Trafﬁcserver-Server Documentation, latest
Troubleshooting Tips
The throughput statistic is inaccurate
Trafﬁc Server updates the throughput statistic after it has transferred an entire object. For larger ﬁles, the byte count increases sharply at the end of a transfer. The complete number of bytes transferred is attributed to the last 10-second interval, although it can take several minutes to transfer the object. This inaccuracy is more noticeable with a light load. A heavier load yields a more accurate statistic.
You are unable to execute Trafﬁc Line commands
Trafﬁc Line commands do not execute under the following conditions: When the trafﬁc_manager process is not running Check to see if the traffic_manager process is running by entering the following command:
 pgrep-l traffic_manager
If the traffic_manager process is not running, then enter the following command from the Trafﬁc Server bin directory to start it:
./traffic_manager
When you are not executing the command from $TSHome/bin If the Trafﬁc Server bin directory is not in your path, then prepend the Trafﬁc Line commands with ./ (for example, ./traffic_ctl -h). When multiple Trafﬁc Server installations are present and you are not executing the Trafﬁc Line command from the active Trafﬁc Server path speciﬁed in ‘‘/etc/trafﬁcserver‘‘
You observe inconsistent behavior when one node obtains an object from another node in the cluster
As part of the initial system preparation, you must synchronize the clocks on all nodes in your cluster. Minor time differences do not cause problems, but differences of more than a few minutes can affect Trafﬁc Server operation. You should run a clock synchronization daemon such as xntpd. To obtain the latest version of xntpd, go to http: //www.eecis.udel.edu/~ntp/.
Web browsers display an error document with a ‘data missing’ message
A message similar to the following might display in web browsers:
Data Missing
This document resulted froma POST operation and has expired from the cache. You can
˓→repost the form data to recreate the document by pressing the Reload button.
This is a Web browser issue and not a problem speciﬁc to (or caused by) Trafﬁc Server. Because Web browsers maintain a separate local cache in memory and/or disk on the client system, messages about documents that have expired from cache refer to the browser’s local cache and not to the Trafﬁc Server cache. There is no Trafﬁc Server message or condition that can cause such messages to appear in a web browser.
5.3. Troubleshooting Tips 511 Apache-Trafﬁcserver-Server Documentation, latest
Trafﬁc Server does not resolve any websites
The browser indicates that it is contacting the host and then times out with the following message:
The document contains no data; Try again later, or contact the server's Administrator.
˓→..
Make sure the system is conﬁgured correctly and that Trafﬁc Server can read the name resolution ﬁle: • Check if the server can resolve DNS lookups by issuing the nslookup command (for example, nslookup www.myhost.com). • Check if the resolv.conf(5) ﬁle contains valid IP addresses for your DNS servers. • On some systems, if the resolv.conf(5) ﬁle is unreadable or has no name server entry, then the operating system uses localhost as a name server. Trafﬁc Server, however, does not use this convention. If you want to use localhost as a name server, then you must add a name server entry for 127.0.0.1 or 0.0.0.0 in the resolv.conf(5) ﬁle. • Check that the Trafﬁc Server user account has permission to read the resolv.conf(5) ﬁle. If it does not, then change the ﬁle permissions to rw-r--r-- (644).
‘Maximum document size exceeded’ message in the system log ﬁle
The following message appears in the system log ﬁle:
WARNING: Maximum document size exceeded
A requested object was larger than the maximum size allowed in the Trafﬁc Server cache, so Trafﬁc Server provided proxy service for the oversized object but did not cache it. To set the object size limit for the cache, modify the :ts:cv:‘proxy.conﬁg.cache.max_doc_size‘ variable in the records.conﬁg ﬁle. If you do not want to limit the size of objects in the cache, then set the document size to 0 (zero).
‘DrainIncomingChannel’ message in the system log ﬁle
The following messages may appear in the system log ﬁle:
Feb 20 23:53:40 louis traffic_manager[4414]: ERROR ==> [drainIncomingChannel] Unknown
˓→message:'GET http://www.telechamada.pt/ HTTP/1.0' Feb 20 23:53:46 louis last message repeated1 time Feb 20 23:53:58 louis traffic_manager[4414]: ERROR ==> [drainIncomingChannel] Unknown
˓→message:'GET http://www.ip.pt/ HTTP/1.0'
These error messages indicate that a browser is sending HTTP requests to one of the Trafﬁc Server cluster ports, either rsport (default port 8088) or mcport (default port 8089). Trafﬁc Server discards these requests. This error does not cause any Trafﬁc Server problems. The misconﬁgured browser must be reconﬁgured to use the correct proxy port. Trafﬁc Server clusters should ideally be conﬁgured to use a separate network interface and cluster on a private subnet, so that client machines have no access to the cluster ports.
‘No cop ﬁle’ message in the system log ﬁle
The following message appears repeatedly in the system log ﬁle: traffic_cop[16056]: encountered"var/trafficserver/no_cop" file...exiting
512 Chapter 5. Appendices Apache-Trafﬁcserver-Server Documentation, latest
The ﬁle var/trafficserver/no_cop acts as an administrative control that instructs the traffic_cop pro- cess to exit immediately without starting traffic_manager or performing any health checks. The no_cop ﬁle prevents Trafﬁc Server from starting automatically when it has been stopped with the option:trafﬁcserver stop com- mand. Without this static control, Trafﬁc Server would restart automatically upon system reboot. The no_cop control keeps Trafﬁc Server off until it is explicitly restarted with:
 trafficserver start
Warning in the system log ﬁle when manually editing vaddrs.conﬁg
If you manually edit vaddrs.config as a non-root user, then Trafﬁc Server issues a warning message in the system log ﬁle similar to the following:
WARNING: interface is ignored: Operation not permitted
You can safely ignore this message as Trafﬁc Server will still apply your conﬁguration edits.
Trafﬁc Server is running but no log ﬁles are created
Trafﬁc Server only writes event log ﬁles when there is information to record. If Trafﬁc Server is idle, then it’s possible that no log ﬁles exist. If Trafﬁc Server is not idle, and you still do not see log ﬁles being generated, verify the following: • Make sure you’re looking in the correct directory. By default, Trafﬁc Server creates log ﬁles in the logs directory. This can be modiﬁed by changing :ts:cv:‘proxy.conﬁg.log.logﬁle_dir‘ in records.config. • Check that the log directory has read/write permissions for the Trafﬁc Server user account. If the log directory does not have the correct permissions, then the traffic_server process will be unable to open or create log ﬁles. • Check that logging is enabled by checking the value of the :ts:cv:‘proxy.conﬁg.log.logging_enabled‘ variable in records.config. • Check that a log format is enabled. In records.config, select the standard or custom format by editing variables in the Logging Conﬁg section.
Trafﬁc Server shows an error indicating too many network connections
By default, Trafﬁc Server supports 8000 network connections. Half of this number is allocated for client connections and the remaining half is for origin server connections. A connection throttle event occurs when either client or origin server connections reach 90% of half the conﬁgured total limit (3600 by default). When a connection throttle event occurs, Trafﬁc Server continues processing all existing connections but will not accept new client connection requests until the connection count falls below the limit. Connection throttle events can occur under the following conditions: Connection Spike Too many client requests (enough to exceed your conﬁgured maximum connections) all reach Trafﬁc Server at the same time. Such events are typically transient and require no corrective action if your connection limits are already conﬁgured appropriately for your Trafﬁc Server and origin resources. Service Overload Client requests are arriving at a rate faster than that which Trafﬁc Server can service them. This may indicate network problems between Trafﬁc Server and origin servers or that Trafﬁc Server may require more memory, CPU, cache disks, or other resources to handle the client load.
5.3. Troubleshooting Tips 513 Apache-Trafﬁcserver-Server Documentation, latest
If necessary, you can adjust the maximum number of connections supported by Trafﬁc Server by editing :ts:cv:‘proxy.conﬁg.net.connections_throttle‘ in records.config.
: Do not increase the connection throttle limit unless the system has adequate memory to handle the client connections required. A system with limited RAM might need a throttle limit lower than the default value. Do not set this variable below the minimum value of 100.
Low memory symptoms
Under heavy load, the Linux kernel can run out of RAM. This low memory condition can cause slow performance and a variety of other system problems. In fact, RAM exhaustion can occur even if the system has plenty of free swap space. Symptoms of extreme memory exhaustion include the following messages in the system log ﬁles (/var/log/ messages):
WARNING: errno 105 is ENOBUFS (low on kernel memory), consider a memory upgrade kernel: eth0: can't fill rx buffer (force 0)! kernel: recvmsg bug: copied E01BA916 seq E01BAB22
To avoid memory exhaustion, add more RAM to the system or reduce the load on Trafﬁc Server.
Conﬁg checker
Trafﬁc Server supports the below command to validate the conﬁg ofﬂine, in order to allow the conﬁg to be pre-checked for possible service disruptions due to synatx errors: traffic_server-Cverify_config-D
 is the location of the conﬁg ﬁles to be validated.
Connection timeouts with the origin server
By default, Trafﬁc Server will timeout after 30 seconds when contacting origin servers. If you cannot avoid such time- outs by otherwise addressing the performance on your origin servers, you may adjust the origin connection timeout in Trafﬁc Server by changing :ts:cv:‘proxy.conﬁg.http.connect_attempts_timeout‘ in records.config to a larger value.
Glossary alternate A variant of a cache object. This was originally created to handle the VARY mechanism but has since been used for additional purposes. All alternates of an object must be equivalent in some manner. That is, they are alternate forms of the same stream. The most common example is having normal and compressed versions of the stream. cache fragment The unit of storage in the cache. All reads from the cache always read exactly one fragment. Fragments may be written in groups, but every write is always an integral number of fragments. Each fragment has a corresponding directory entry which describes its location in the cache storage.
514 Chapter 5. Appendices Apache-Trafﬁcserver-Server Documentation, latest cache ID A 128 bit value used as a ﬁxed sized identiﬁer for an object in the cache. This is computed from the cache key using the MD5 hashing function. cache key A byte sequence that is a globally unique identiﬁer for an object in the cache. By default the URL for the object is used. cache object The minimal self contained unit of data in the cache. Cache objects are the stored version of equivalent content streams from an origin server. A single object can have multiple variants called alternates. cache partition A subdivision of the cache storage in Trafﬁc Server which is dedicated to objects for speciﬁc proto- cols, origins, or other rules. Deﬁning and managing cache partitions is discussed in Partitioning the Cache. cache span The physical storage described by a single line in storage.config. cache stripe A homogenous, persistent store for the cache in a single cache span. A stripe always resides entirely on a single physical device and is treated as an undifferentiated span of bytes. This is the smallest independent unit of storage. cache tag The bottom few bits (12 currently) of the cache ID. This is used in the cache directory for a preliminary identity check before going to disk. cache volume A user deﬁned unit of persistent storage for the cache. Cache volumes are deﬁned in volume. config. A cache volume is by default spread across cache spans to increase robustness. Each section of a cache volume on a speciﬁc cache span is a cache stripe. continuation A callable object that contains state. These are are mechanism used by Trafﬁc Server to implement callbacks and continued computations. Continued computations are critical to efﬁcient processing of trafﬁc because by avoiding any blocking operations that wait on external events. In any such case a continuation is used so that other processing can continue until the external event occurs. At that point the continuation is invoked to continue the suspended processing. This can be considered similar to co-routines. directory bucket A contiguous ﬁxed sized group of directory entries. This is used for hash bucket maintenance optimization. directory entry An in memory entry that describes a cache fragment. directory segment A contiguous group of buckets. Each cache stripe has a set of segments all of which have the same number of buckets, although the number of buckets per segment can vary between cache stripes. Segments are administrative in purpose to minimize the size of free list and hash bucket pointers. fresh The state of a cache object which can be served directly from the the cache in response to client requests. Fresh objects have not met or passed their origin server deﬁned expiration time, nor have they reached the algorithmically determined stale age. global plugin A plugin which operates on all transactions. Contrast with remap plugin. object store The database of cache objects. origin server An HTTP server which provides the original source of content being cached by Trafﬁc Server. remap plugin A plugin which operates only on transactions matching speciﬁc remap rules as deﬁned in remap. config. Contrast with global plugin. revalidation Verifying that a currently cached object is still valid. This is usually done using an If-Modiﬁed-Since request which allows the origin server to validate the content without resending the content. session A single connection from a client to Trafﬁc Server, covering all requests and responses on that connection. A session starts when the client connection opens, and ends when the connection closes. stale The state of a cache object which is not yet expired, but has reached an algorithmically determined age at which the origin server will be contacted to revalidate the freshness of the object. Contrast with fresh. storage unit Obsolete term for cache span.
5.4. Glossary 515 Apache-Trafﬁcserver-Server Documentation, latest transaction A client request and response, either from the origin server or from the cache. A transaction begins when Trafﬁc Server receives a request, and ends when Trafﬁc Server sends the response. write cursor The location in a cache stripe where new data is written.
HTTP Status Codes
The following standard HTTP response messages are provided for your information. Code Description 200 OK 202 Accepted 204 No Content 206 Partial Content 300 Multiple Choices 301 Moved Permanently 302 Found 303 See Other 304 Not Modiﬁed 400 Bad Request 401 Unauthorized; retry 403 Forbidden 404 Not Found 405 Method Not Allowed 406 Not Acceptable 408 Request Timeout 500 Internal Server Error 501 Not Implemented 502 Bad Gateway 504 Gateway Timeout More detail can be found in RFC2616 which deﬁnes the full range of standardized HTTP/1.1 response codes.
516 Chapter 5. Appendices CHAPTER 6
Indices and tables
• genindex • search
517 Apache-Trafﬁcserver-Server Documentation, latest
518 Chapter 6. Indices and tables A Doc::Doc::len (C++ ), 284 AFTER_BODY (C ), 384 Doc::Doc::magic (C++ ), 284 AFTER_HEADER (C ), 384 Doc::Doc::sync_serial (C++ ), 285 alternate, 272, 514 Doc::Doc::total_len (C++ ), 284 Doc::Doc::write_serial (C++ ), 285 C Doc::pinned (C++ ), 285 cache directory, 269 E cache fragment, 514 cache ID, 269, 515 EvacuationBlock (C++ ), 284 cache key, 515 F cache object, 515 cache partition, 515 fragment, 269 cache span, 515 fresh, 515 cache stripe, 515 cache tag, 515 G cache volume, 515 global plugin, 515 CacheControlResult (C++ ), 283 CacheHTTPInfo (C++ ), 283 H CacheHTTPInfoVector (C++ ), 283 HTTPCacheAlt (C++ ), 283 CacheVC (C++ ), 283 HTTPInfo (C++ ), 283 CacheVC::openReadStartEarliest (C++ ), 283 HttpTunnel (C++ ), 283 CacheVC::openReadStartHead (C++ ), 283 CacheVol (C++ ), 284 I continuation, 515 index buckets, 276 index segment, 276 D ink_hrtime (C ), 459 DEPRECATED (C ), 390 Dir (C++ ), 275 N DIR_BLOCK_SIZE (C++ ), 285 NO_CALLBACK (C ), 384 DIR_DEPTH, 276 DIR_SIZE_WITH_BLOCK (C++ ), 285 O directory bucket, 515 object store, 515 directory entry, 269, 515 OpenDir (C++ ), 283 directory segment, 515 origin server, 515 Doc (C++ ), 284 Doc::checksum (C++ ), 285 P Doc::Doc::ﬁrst_key (C++ ), 284 pinned, 274 Doc::Doc::ﬂen (C++ ), 284 port descriptors, 237 Doc::Doc::ftype (C++ ), 284 Doc::Doc::hlen (C++ ), 284 R Doc::Doc::key (C++ ), 284 remap plugin, 515
519 Apache-Trafﬁcserver-Server Documentation, latest revalidation, 515 TS_CONFIG_HTTP_CACHE_FUZZ_MIN_TIME (C ), RFC 392 RFC 6961, 58 TS_CONFIG_HTTP_CACHE_FUZZ_PROBABILITY (C ), 392 S TS_CONFIG_HTTP_CACHE_FUZZ_TIME (C ), 392 session, 515 TS_CONFIG_HTTP_CACHE_GUARANTEED_MAX_LIFETIME stale, 515 (C ), 391 storage unit, 515 TS_CONFIG_HTTP_CACHE_GUARANTEED_MIN_LIFETIME (C ), 391 T TS_CONFIG_HTTP_CACHE_HEURISTIC_LM_FACTOR transaction, 516 (C ), 392 TS_AIO_EVENT_DONE (C ), 383 TS_CONFIG_HTTP_CACHE_HEURISTIC_MAX_LIFETIME TS_CACHE_DATA_TYPE_HTTP (C ), 380 (C ), 391 TS_CACHE_DATA_TYPE_NONE (C ), 380 TS_CONFIG_HTTP_CACHE_HEURISTIC_MIN_LIFETIME TS_CACHE_DATA_TYPE_OTHER (C ), 380 (C ), 391 TS_CACHE_ERROR_DOC_BUSY (C ), 380 TS_CONFIG_HTTP_CACHE_HTTP (C ), 391 TS_CACHE_ERROR_NO_DOC (C ), 380 TS_CONFIG_HTTP_CACHE_IGNORE_AUTHENTICATION TS_CACHE_ERROR_NOT_READY (C ), 380 (C ), 391 TS_CACHE_LOOKUP_HIT_FRESH (C ), 381 TS_CONFIG_HTTP_CACHE_IGNORE_CLIENT_CC_MAX_AGE TS_CACHE_LOOKUP_HIT_STALE (C ), 381 (C ), 391 TS_CACHE_LOOKUP_MISS (C ), 381 TS_CONFIG_HTTP_CACHE_IGNORE_CLIENT_NO_CACHE TS_CACHE_LOOKUP_SKIPPED (C ), 381 (C ), 391 TS_CACHE_SCAN_RESULT_CONTINUE (C ), 381 TS_CONFIG_HTTP_CACHE_IGNORE_SERVER_NO_CACHE TS_CACHE_SCAN_RESULT_DELETE (C ), 381 (C ), 391 TS_CACHE_SCAN_RESULT_DELETE_ALL_ALTERNATESTS_CONFIG_HTTP_CACHE_IMS_ON_CLIENT_NO_CACHE (C ), 381 (C ), 391 TS_CACHE_SCAN_RESULT_DONE (C ), 381 TS_CONFIG_HTTP_CACHE_MAX_OPEN_READ_RETRIES TS_CACHE_SCAN_RESULT_RETRY (C ), 381 (C ), 392 TS_CACHE_SCAN_RESULT_UPDATE (C ), 381 TS_CONFIG_HTTP_CACHE_MAX_STALE_AGE (C TS_CONFIG_HTTP_ACCEPT_ENCODING_FILTER_ENABLED ), 391 (C ), 392 TS_CONFIG_HTTP_CACHE_OPEN_READ_RETRY_TIME TS_CONFIG_HTTP_ANONYMIZE_INSERT_CLIENT_IP (C ), 392 (C ), 391 TS_CONFIG_HTTP_CACHE_RANGE_LOOKUP (C ), TS_CONFIG_HTTP_ANONYMIZE_REMOVE_CLIENT_IP 392 (C ), 391 TS_CONFIG_HTTP_CACHE_RANGE_WRITE (C ), TS_CONFIG_HTTP_ANONYMIZE_REMOVE_COOKIE 392 (C ), 391 TS_CONFIG_HTTP_CACHE_REQUIRED_HEADERS TS_CONFIG_HTTP_ANONYMIZE_REMOVE_FROM (C ), 391 (C ), 391 TS_CONFIG_HTTP_CACHE_WHEN_TO_REVALIDATE TS_CONFIG_HTTP_ANONYMIZE_REMOVE_REFERER (C ), 390 (C ), 391 TS_CONFIG_HTTP_CHUNKING_ENABLED (C ), TS_CONFIG_HTTP_ANONYMIZE_REMOVE_USER_AGENT 390 (C ), 391 TS_CONFIG_HTTP_CHUNKING_SIZE (C ), 392 TS_CONFIG_HTTP_BACKGROUND_FILL_ACTIVE_TIMEOUTTS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES (C ), 392 (C ), 391 TS_CONFIG_HTTP_BACKGROUND_FILL_COMPLETED_THRESHOLDTS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_DEAD_SERVER (C ), 392 (C ), 391 TS_CONFIG_HTTP_CACHE_CACHE_RESPONSES_TO_COOKIESTS_CONFIG_HTTP_CONNECT_ATTEMPTS_RR_RETRIES (C ), 391 (C ), 391 TS_CONFIG_HTTP_CACHE_CACHE_URLS_THAT_LOOK_DYNAMICTS_CONFIG_HTTP_CONNECT_ATTEMPTS_TIMEOUT (C ), 391 (C ), 392 TS_CONFIG_HTTP_CACHE_CLUSTER_CACHE_LOCALTS_CONFIG_HTTP_DEFAULT_BUFFER_SIZE (C ), (C ), 391 392 TS_CONFIG_HTTP_DEFAULT_BUFFER_WATER_MARK
520 Apache-Trafﬁcserver-Server Documentation, latest
(C ), 392 TS_CONFIG_HTTP_RESPONSE_SERVER_ENABLED TS_CONFIG_HTTP_DOC_IN_CACHE_SKIP_DNS (C (C ), 391 ), 392 TS_CONFIG_HTTP_RESPONSE_SERVER_STR (C ), TS_CONFIG_HTTP_DOWN_SERVER_ABORT_THRESHOLD 392 (C ), 392 TS_CONFIG_HTTP_SEND_HTTP11_REQUESTS (C TS_CONFIG_HTTP_DOWN_SERVER_CACHE_TIME ), 391 (C ), 392 TS_CONFIG_HTTP_SERVER_SESSION_SHARING_MATCH TS_CONFIG_HTTP_FLOW_CONTROL_ENABLED (C ), 390 (C ), 392 TS_CONFIG_HTTP_SERVER_SESSION_SHARING_POOL TS_CONFIG_HTTP_FLOW_CONTROL_HIGH_WATER_MARK (C ), 390 (C ), 392 TS_CONFIG_HTTP_SERVER_TCP_INIT_CWND (C TS_CONFIG_HTTP_FLOW_CONTROL_LOW_WATER_MARK ), 391 (C ), 392 TS_CONFIG_HTTP_TRANSACTION_ACTIVE_TIMEOUT_OUT TS_CONFIG_HTTP_FORWARD_PROXY_AUTH_TO_PARENT (C ), 391 (C ), 391 TS_CONFIG_HTTP_TRANSACTION_NO_ACTIVITY_TIMEOUT_IN TS_CONFIG_HTTP_GLOBAL_USER_AGENT_HEADER (C ), 391 (C ), 392 TS_CONFIG_HTTP_TRANSACTION_NO_ACTIVITY_TIMEOUT_OUT TS_CONFIG_HTTP_INSERT_AGE_IN_RESPONSE (C ), 391 (C ), 392 TS_CONFIG_LAST_ENTRY (C ), 392 TS_CONFIG_HTTP_INSERT_REQUEST_VIA_STR TS_CONFIG_NET_SOCK_OPTION_FLAG_OUT (C ), (C ), 391 390 TS_CONFIG_HTTP_INSERT_RESPONSE_VIA_STR TS_CONFIG_NET_SOCK_PACKET_MARK_OUT (C (C ), 391 ), 392 TS_CONFIG_HTTP_INSERT_SQUID_X_FORWARDED_FORTS_CONFIG_NET_SOCK_PACKET_TOS_OUT (C ), (C ), 391 392 TS_CONFIG_HTTP_KEEP_ALIVE_ENABLED_IN (C TS_CONFIG_NET_SOCK_RECV_BUFFER_SIZE_OUT ), 390 (C ), 390 TS_CONFIG_HTTP_KEEP_ALIVE_ENABLED_OUT TS_CONFIG_NET_SOCK_SEND_BUFFER_SIZE_OUT (C ), 390 (C ), 390 TS_CONFIG_HTTP_KEEP_ALIVE_NO_ACTIVITY_TIMEOUT_INTS_CONFIG_NULL (C ), 390 (C ), 391 TS_CONFIG_SSL_HSTS_INCLUDE_SUBDOMAINS TS_CONFIG_HTTP_KEEP_ALIVE_NO_ACTIVITY_TIMEOUT_OUT(C ), 392 (C ), 391 TS_CONFIG_SSL_HSTS_MAX_AGE (C ), 392 TS_CONFIG_HTTP_KEEP_ALIVE_POST_OUT (C ), TS_CONFIG_URL_REMAP_PRISTINE_HOST_HDR 390 (C ), 390 TS_CONFIG_HTTP_NEGATIVE_CACHING_ENABLEDTS_ERROR (C ), 396 (C ), 390 TS_EVENT_CACHE_CLOSE (C ), 383 TS_CONFIG_HTTP_NEGATIVE_CACHING_LIFETIMETS_EVENT_CACHE_DELETE (C ), 382 (C ), 390 TS_EVENT_CACHE_LOOKUP (C ), 382 TS_CONFIG_HTTP_NEGATIVE_REVALIDATING_ENABLEDTS_EVENT_CACHE_LOOKUP_COMPLETE (C ), 383 (C ), 392 TS_EVENT_CACHE_LOOKUP_READY (C ), 383 TS_CONFIG_HTTP_NEGATIVE_REVALIDATING_LIFETIMETS_EVENT_CACHE_OPEN_READ (C ), 382 (C ), 392 TS_EVENT_CACHE_OPEN_READ_FAILED (C ), 382 TS_CONFIG_HTTP_NORMALIZE_AE_GZIP (C ), 392 TS_EVENT_CACHE_OPEN_WRITE (C ), 382 TS_CONFIG_HTTP_ORIGIN_MAX_CONNECTIONS TS_EVENT_CACHE_OPEN_WRITE_FAILED (C ), (C ), 391 382 TS_CONFIG_HTTP_POST_CHECK_CONTENT_LENGTH_ENABLEDTS_EVENT_CACHE_READ (C ), 382 (C ), 392 TS_EVENT_CACHE_READ_COMPLETE (C ), 383 TS_CONFIG_HTTP_POST_CONNECT_ATTEMPTS_TIMEOUTTS_EVENT_CACHE_READ_READY (C ), 383 (C ), 392 TS_EVENT_CACHE_REMOVE (C ), 382 TS_CONFIG_HTTP_REQUEST_HEADER_MAX_SIZE TS_EVENT_CACHE_REMOVE_FAILED (C ), 382 (C ), 392 TS_EVENT_CACHE_SCAN (C ), 382 TS_CONFIG_HTTP_RESPONSE_HEADER_MAX_SIZETS_EVENT_CACHE_SCAN_DONE (C ), 382 (C ), 392 TS_EVENT_CACHE_SCAN_FAILED (C ), 382
521 Apache-Trafﬁcserver-Server Documentation, latest
TS_EVENT_CACHE_SCAN_OBJECT (C ), 382 TS_EVENT_NET_CONNECT (C ), 382 TS_EVENT_CACHE_SCAN_OPERATION_BLOCKED TS_EVENT_NET_CONNECT_FAILED (C ), 382 (C ), 382 TS_EVENT_NONE (C ), 381 TS_EVENT_CACHE_SCAN_OPERATION_FAILED TS_EVENT_TIMEOUT (C ), 381 (C ), 382 TS_EVENT_VCONN_ACTIVE_TIMEOUT (C ), 382 TS_EVENT_CACHE_WRITE (C ), 382 TS_EVENT_VCONN_EOS (C ), 382 TS_EVENT_CACHE_WRITE_HEADER (C ), 383 TS_EVENT_VCONN_INACTIVITY_TIMEOUT (C ), TS_EVENT_CONTINUE (C ), 382 382 TS_EVENT_ERROR (C ), 381 TS_EVENT_VCONN_PRE_ACCEPT (C ), 383 TS_EVENT_HOST_LOOKUP (C ), 382 TS_EVENT_VCONN_READ_COMPLETE (C ), 382 TS_EVENT_HTTP_CACHE_LOOKUP_COMPLETE TS_EVENT_VCONN_READ_READY (C ), 382 (C ), 383 TS_EVENT_VCONN_WRITE_COMPLETE (C ), 382 TS_EVENT_HTTP_CONTINUE (C ), 383 TS_EVENT_VCONN_WRITE_READY (C ), 382 TS_EVENT_HTTP_ERROR (C ), 383 TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK (C TS_EVENT_HTTP_OS_DNS (C ), 383 ), 385 TS_EVENT_HTTP_POST_REMAP (C ), 383 TS_HTTP_LAST_HOOK (C ), 385 TS_EVENT_HTTP_PRE_REMAP (C ), 383 TS_HTTP_OS_DNS_HOOK (C ), 384 TS_EVENT_HTTP_READ_CACHE_HDR (C ), 383 TS_HTTP_POST_REMAP_HOOK (C ), 385 TS_EVENT_HTTP_READ_REQUEST_HDR (C ), 383 TS_HTTP_PRE_REMAP_HOOK (C ), 385 TS_EVENT_HTTP_READ_RESPONSE_HDR (C ), 383 TS_HTTP_READ_CACHE_HDR_HOOK (C ), 384 TS_EVENT_HTTP_REQUEST_TRANSFORM (C ), TS_HTTP_READ_REQUEST_HDR_HOOK (C ), 384 383 TS_HTTP_READ_RESPONSE_HDR_HOOK (C ), 384 TS_EVENT_HTTP_RESPONSE_TRANSFORM (C ), TS_HTTP_REQUEST_TRANSFORM_HOOK (C ), 384 383 TS_HTTP_RESPONSE_CLIENT_HOOK (C ), 385 TS_EVENT_HTTP_SELECT_ALT (C ), 383 TS_HTTP_RESPONSE_TRANSFORM_HOOK (C ), TS_EVENT_HTTP_SEND_REQUEST_HDR (C ), 383 384 TS_EVENT_HTTP_SEND_RESPONSE_HDR (C ), 383 TS_HTTP_SELECT_ALT_HOOK (C ), 384 TS_EVENT_HTTP_SSN_CLOSE (C ), 383 TS_HTTP_SEND_REQUEST_HDR_HOOK (C ), 384 TS_EVENT_HTTP_SSN_START (C ), 383 TS_HTTP_SEND_RESPONSE_HDR_HOOK (C ), 384 TS_EVENT_HTTP_TXN_CLOSE (C ), 383 TS_HTTP_SSN_CLOSE_HOOK (C ), 385 TS_EVENT_HTTP_TXN_START (C ), 383 TS_HTTP_SSN_START_HOOK (C ), 384 TS_EVENT_IMMEDIATE (C ), 381 TS_HTTP_STATUS_ACCEPTED (C ), 385 TS_EVENT_INTERNAL_1200 (C ), 383 TS_HTTP_STATUS_ALREADY_REPORTED (C ), 385 TS_EVENT_INTERNAL_206 (C ), 382 TS_HTTP_STATUS_BAD_GATEWAY (C ), 386 TS_EVENT_INTERNAL_207 (C ), 382 TS_HTTP_STATUS_BAD_REQUEST (C ), 386 TS_EVENT_INTERNAL_208 (C ), 382 TS_HTTP_STATUS_CONFLICT (C ), 386 TS_EVENT_INTERNAL_209 (C ), 382 TS_HTTP_STATUS_CONTINUE (C ), 385 TS_EVENT_INTERNAL_210 (C ), 382 TS_HTTP_STATUS_CREATED (C ), 385 TS_EVENT_INTERNAL_211 (C ), 382 TS_HTTP_STATUS_EXPECTATION_FAILED (C ), TS_EVENT_INTERNAL_212 (C ), 382 386 TS_EVENT_INTERNAL_60200 (C ), 383 TS_HTTP_STATUS_FAILED_DEPENDENCY (C ), TS_EVENT_INTERNAL_60201 (C ), 383 386 TS_EVENT_INTERNAL_60202 (C ), 383 TS_HTTP_STATUS_FORBIDDEN (C ), 386 TS_EVENT_LIFECYCLE_CACHE_READY (C ), 383 TS_HTTP_STATUS_GATEWAY_TIMEOUT (C ), 386 TS_EVENT_LIFECYCLE_CLIENT_SSL_CTX_INITIALIZEDTS_HTTP_STATUS_GONE (C ), 386 (C ), 383 TS_HTTP_STATUS_HTTPVER_NOT_SUPPORTED TS_EVENT_LIFECYCLE_PORTS_INITIALIZED (C ), (C ), 387 383 TS_HTTP_STATUS_IM_USED (C ), 385 TS_EVENT_LIFECYCLE_PORTS_READY (C ), 383 TS_HTTP_STATUS_INSUFFICIENT_STORAGE (C ), TS_EVENT_LIFECYCLE_SERVER_SSL_CTX_INITIALIZED 387 (C ), 383 TS_HTTP_STATUS_INTERNAL_SERVER_ERROR (C TS_EVENT_MGMT_UPDATE (C ), 383 ), 386 TS_EVENT_NET_ACCEPT (C ), 382 TS_HTTP_STATUS_LENGTH_REQUIRED (C ), 386 TS_EVENT_NET_ACCEPT_FAILED (C ), 382 TS_HTTP_STATUS_LOCKED (C ), 386
522 Apache-Trafﬁcserver-Server Documentation, latest
TS_HTTP_STATUS_LOOP_DETECTED (C ), 387 (C ), 386 TS_HTTP_STATUS_METHOD_NOT_ALLOWED (C ), TS_HTTP_STATUS_UPGRADE_REQUIRED (C ), 386 386 TS_HTTP_STATUS_USE_PROXY (C ), 386 TS_HTTP_STATUS_MOVED_PERMANENTLY (C ), TS_HTTP_STATUS_VARIANT_ALSO_NEGOTIATES 386 (C ), 387 TS_HTTP_STATUS_MOVED_TEMPORARILY (C ), TS_HTTP_TXN_CLOSE_HOOK (C ), 384 386 TS_HTTP_TXN_START_HOOK (C ), 384 TS_HTTP_STATUS_MULTI_STATUS (C ), 385 TS_HTTP_TYPE_REQUEST (C ), 387 TS_HTTP_STATUS_MULTIPLE_CHOICES (C ), 385 TS_HTTP_TYPE_RESPONSE (C ), 387 TS_HTTP_STATUS_NETWORK_AUTHENTICATION_REQUIREDTS_HTTP_TYPE_UNKNOWN (C ), 387 (C ), 387 TS_IOBUFFER_SIZE_INDEX_128 (C ), 388 TS_HTTP_STATUS_NO_CONTENT (C ), 385 TS_IOBUFFER_SIZE_INDEX_128K (C ), 388 TS_HTTP_STATUS_NON_AUTHORITATIVE_INFORMATIONTS_IOBUFFER_SIZE_INDEX_16K (C ), 388 (C ), 385 TS_IOBUFFER_SIZE_INDEX_1K (C ), 388 TS_HTTP_STATUS_NONE (C ), 385 TS_IOBUFFER_SIZE_INDEX_1M (C ), 388 TS_HTTP_STATUS_NOT_ACCEPTABLE (C ), 386 TS_IOBUFFER_SIZE_INDEX_256 (C ), 388 TS_HTTP_STATUS_NOT_EXTENDED (C ), 387 TS_IOBUFFER_SIZE_INDEX_256K (C ), 388 TS_HTTP_STATUS_NOT_FOUND (C ), 386 TS_IOBUFFER_SIZE_INDEX_2K (C ), 388 TS_HTTP_STATUS_NOT_IMPLEMENTED (C ), 386 TS_IOBUFFER_SIZE_INDEX_2M (C ), 388 TS_HTTP_STATUS_NOT_MODIFIED (C ), 386 TS_IOBUFFER_SIZE_INDEX_32K (C ), 388 TS_HTTP_STATUS_OK (C ), 385 TS_IOBUFFER_SIZE_INDEX_4K (C ), 388 TS_HTTP_STATUS_PARTIAL_CONTENT (C ), 385 TS_IOBUFFER_SIZE_INDEX_512 (C ), 388 TS_HTTP_STATUS_PAYMENT_REQUIRED (C ), 386 TS_IOBUFFER_SIZE_INDEX_512K (C ), 388 TS_HTTP_STATUS_PERMANENT_REDIRECT (C ), TS_IOBUFFER_SIZE_INDEX_64K (C ), 388 386 TS_IOBUFFER_SIZE_INDEX_8K (C ), 388 TS_HTTP_STATUS_PRECONDITION_FAILED (C ), TS_LIFECYCLE_CACHE_READY_HOOK (C ), 436 386 TS_LIFECYCLE_CACHE_READY_HOOK (C ), 388 TS_HTTP_STATUS_PRECONDITION_REQUIRED (C TS_LIFECYCLE_CLIENT_SSL_CTX_INITIALIZED_HOOK ), 386 (C ), 388 TS_HTTP_STATUS_PROXY_AUTHENTICATION_REQUIREDTS_LIFECYCLE_LAST_HOOK (C ), 388 (C ), 386 TS_LIFECYCLE_PORTS_INITIALIZED_HOOK (C ), TS_HTTP_STATUS_REQUEST_ENTITY_TOO_LARGE 436 (C ), 386 TS_LIFECYCLE_PORTS_INITIALIZED_HOOK (C ), TS_HTTP_STATUS_REQUEST_HEADER_FIELDS_TOO_LARGE388 (C ), 386 TS_LIFECYCLE_PORTS_READY_HOOK (C ), 436 TS_HTTP_STATUS_REQUEST_TIMEOUT (C ), 386 TS_LIFECYCLE_PORTS_READY_HOOK (C ), 388 TS_HTTP_STATUS_REQUEST_URI_TOO_LONG (C TS_LIFECYCLE_SERVER_SSL_CTX_INITIALIZED_HOOK ), 386 (C ), 388 TS_HTTP_STATUS_REQUESTED_RANGE_NOT_SATISFIABLETS_LOOKUP_HOST_NONE (C ), 389 (C ), 386 TS_LOOKUP_ICP_SUGGESTED_HOST (C ), 389 TS_HTTP_STATUS_RESET_CONTENT (C ), 385 TS_LOOKUP_INCOMING_ROUTER (C ), 389 TS_HTTP_STATUS_SEE_OTHER (C ), 386 TS_LOOKUP_ORIGIN_SERVER (C ), 389 TS_HTTP_STATUS_SERVICE_UNAVAILABLE (C ), TS_LOOKUP_PARENT_PROXY (C ), 389 386 TS_LOOKUP_UNDEFINED_LOOKUP (C ), 389 TS_HTTP_STATUS_SWITCHING_PROTOCOL (C ), TS_MILESTONE_CACHE_OPEN_READ_BEGIN (C 385 ), 390 TS_HTTP_STATUS_TEMPORARY_REDIRECT (C ), TS_MILESTONE_CACHE_OPEN_READ_END (C ), 386 390 TS_HTTP_STATUS_TOO_MANY_REQUESTS (C ), TS_MILESTONE_CACHE_OPEN_WRITE_BEGIN (C 386 ), 390 TS_HTTP_STATUS_UNAUTHORIZED (C ), 386 TS_MILESTONE_CACHE_OPEN_WRITE_END (C ), TS_HTTP_STATUS_UNPROCESSABLE_ENTITY (C 390 ), 386 TS_MILESTONE_DNS_LOOKUP_BEGIN (C ), 390 TS_HTTP_STATUS_UNSUPPORTED_MEDIA_TYPE TS_MILESTONE_DNS_LOOKUP_END (C ), 390
523 Apache-Trafﬁcserver-Server Documentation, latest
TS_MILESTONE_LAST_ENTRY (C ), 390 TS_RECORDUPDATE_NULL (C ), 396 TS_MILESTONE_NULL (C ), 389 TS_RECORDUPDATE_RESTART_TC (C ), 396 TS_MILESTONE_SERVER_BEGIN_WRITE (C ), 389 TS_RECORDUPDATE_RESTART_TM (C ), 396 TS_MILESTONE_SERVER_CLOSE (C ), 390 TS_RECORDUPDATE_RESTART_TS (C ), 396 TS_MILESTONE_SERVER_CONNECT (C ), 389 TS_SDK_VERSION_2_0 (C ), 397 TS_MILESTONE_SERVER_CONNECT_END (C ), TS_SDK_VERSION_3_0 (C ), 397 389 TS_SERVER_SESSION_SHARING_MATCH_BOTH TS_MILESTONE_SERVER_FIRST_CONNECT (C ), (C ), 397 389 TS_SERVER_SESSION_SHARING_MATCH_HOST TS_MILESTONE_SERVER_FIRST_READ (C ), 389 (C ), 397 TS_MILESTONE_SERVER_READ_HEADER_DONE TS_SERVER_SESSION_SHARING_MATCH_IP (C ), (C ), 389 397 TS_MILESTONE_SM_FINISH (C ), 390 TS_SERVER_SESSION_SHARING_MATCH_NONE TS_MILESTONE_SM_START (C ), 390 (C ), 397 TS_MILESTONE_UA_BEGIN (C ), 389 TS_SERVER_SESSION_SHARING_POOL_GLOBAL TS_MILESTONE_UA_BEGIN_WRITE (C ), 389 (C ), 397 TS_MILESTONE_UA_CLOSE (C ), 389 TS_SERVER_SESSION_SHARING_POOL_THREAD TS_MILESTONE_UA_READ_HEADER_DONE (C ), (C ), 397 389 TS_SRVSTATE_ACTIVE_TIMEOUT (C ), 398 TS_PARSE_CONT (C ), 393 TS_SRVSTATE_BAD_INCOMING_RESPONSE (C ), TS_PARSE_DONE (C ), 393 398 TS_PARSE_ERROR (C ), 393 TS_SRVSTATE_CONGEST_CONTROL_CONGESTED_ON_F TS_PARSE_OK (C ), 393 (C ), 398 TS_RECORDACCESS_NO_ACCESS (C ), 393 TS_SRVSTATE_CONGEST_CONTROL_CONGESTED_ON_M TS_RECORDACCESS_NULL (C ), 393 (C ), 398 TS_RECORDACCESS_READ_ONLY (C ), 393 TS_SRVSTATE_CONNECTION_ALIVE (C ), 398 TS_RECORDCHECK_INT (C ), 394 TS_SRVSTATE_CONNECTION_CLOSED (C ), 398 TS_RECORDCHECK_IP (C ), 394 TS_SRVSTATE_CONNECTION_ERROR (C ), 398 TS_RECORDCHECK_NULL (C ), 394 TS_SRVSTATE_INACTIVE_TIMEOUT (C ), 398 TS_RECORDCHECK_STR (C ), 394 TS_SRVSTATE_OPEN_RAW_ERROR (C ), 398 TS_RECORDDATATYPE_COUNTER (C ), 394 TS_SRVSTATE_PARSE_ERROR (C ), 398 TS_RECORDDATATYPE_FLOAT (C ), 394 TS_SRVSTATE_STATE_UNDEFINED (C ), 398 TS_RECORDDATATYPE_INT (C ), 394 TS_SRVSTATE_TRANSACTION_COMPLETE (C ), TS_RECORDDATATYPE_MAX (C ), 394 398 TS_RECORDDATATYPE_NULL (C ), 394 TS_SSL_CERT_HOOK (C ), 385 TS_RECORDDATATYPE_STAT_CONST (C ), 394 TS_SSL_FIRST_HOOK (C ), 385 TS_RECORDDATATYPE_STAT_FX (C ), 394 TS_SSL_HOOK_OP_DEFAULT (C ), 398 TS_RECORDDATATYPE_STRING (C ), 394 TS_SSL_HOOK_OP_LAST (C ), 398 TS_RECORDMODE_CLIENT (C ), 395 TS_SSL_HOOK_OP_TERMINATE (C ), 398 TS_RECORDMODE_NULL (C ), 395 TS_SSL_HOOK_OP_TUNNEL (C ), 398 TS_RECORDMODE_SERVER (C ), 395 TS_SSL_LAST_HOOK (C ), 385 TS_RECORDMODE_STAND_ALONE (C ), 395 TS_SSL_SNI_HOOK (C ), 385 TS_RECORDP_NON_PERSISTENT (C ), 395 TS_SUCCESS (C ), 396 TS_RECORDP_NULL (C ), 395 TS_THREAD_POOL_CLUSTER (C ), 399 TS_RECORDP_PERSISTENT (C ), 395 TS_THREAD_POOL_DEFAULT (C ), 399 TS_RECORDTYPE_ALL (C ), 396 TS_THREAD_POOL_DNS (C ), 399 TS_RECORDTYPE_CLUSTER (C ), 395 TS_THREAD_POOL_NET (C ), 399 TS_RECORDTYPE_CONFIG (C ), 395 TS_THREAD_POOL_REMAP (C ), 399 TS_RECORDTYPE_LOCAL (C ), 395 TS_THREAD_POOL_SSL (C ), 399 TS_RECORDTYPE_NODE (C ), 395 TS_THREAD_POOL_TASK (C ), 399 TS_RECORDTYPE_NULL (C ), 395 TS_THREAD_POOL_UDP (C ), 399 TS_RECORDTYPE_PLUGIN (C ), 395 TS_VC_CLOSE_ABORT (C ), 399 TS_RECORDTYPE_PROCESS (C ), 395 TS_VC_CLOSE_NORMAL (C ), 399 TS_RECORDUPDATE_DYNAMIC (C ), 396 TS_VCONN_PRE_ACCEPT_HOOK (C ), 385
524 Apache-Trafﬁcserver-Server Documentation, latest
TSAction (C ), 459 TSHttpHdrHostGet (C ), 410 TSActionCancel (C ), 401 TSHttpHdrLengthGet (C ), 410 TSActionDone (C ), 402 TSHttpHdrMethodGet (C ), 410 TSAssert (C ), 406 TSHttpHdrMethodSet (C ), 410 TSCacheDataType (C ), 380 TSHttpHdrParseReq (C ), 419 TSCacheError (C ), 380 TSHttpHdrParseResp (C ), 419 TSCacheKey (C ), 459 TSHttpHdrPrint (C ), 411 TSCacheLookupResult (C ), 380 TSHttpHdrReasonGet (C ), 411 TSCacheRead (C ), 402 TSHttpHdrReasonLookup (C ), 411 TSCacheRemove (C ), 402 TSHttpHdrReasonSet (C ), 411 TSCacheScanResult (C ), 381 TSHttpHdrStatusGet (C ), 411 TSCacheUrlSet (C ), 285 TSHttpHdrStatusSet (C ), 412 TSCacheWrite (C ), 403 TSHttpHdrTypeGet (C ), 412 TSConﬁg (C ), 459 TSHttpHdrTypeSet (C ), 412 TSConﬁgDataGet (C ), 403 TSHttpHdrUrlGet (C ), 412 TSConﬁgDestroyFunc (C ), 459 TSHttpHdrUrlSet (C ), 412 TSConﬁgDirGet (C ), 435 TSHttpHdrVersionGet (C ), 413 TSConﬁgGet (C ), 403 TSHttpHdrVersionSet (C ), 413 TSConﬁgRelease (C ), 403 TSHttpHookAdd (C ), 413 TSConﬁgSet (C ), 404 TSHttpHookID (C ), 384, 460 TSCont (C ), 459 TSHttpHookNameLookup (C ), 406 TSContCall (C ), 404 TSHttpIsInternalRequest (C ), 415 TSContCreate (C ), 404 TSHttpIsInternalSession (C ), 415 TSContDataGet (C ), 404 TSHttpParser (C ), 460 TSContDataSet (C ), 404 TSHttpParserClear (C ), 418 TSContDestroy (C ), 405 TSHttpParserCreate (C ), 418 TSContMutexGet (C ), 405 TSHttpParserDestroy (C ), 419 TSContSchedule (C ), 405 TSHttpServerStateNameLookup (C ), 406 TSDebug (C ), 405 TSHttpSsn (C ), 460 TSDebugSpeciﬁc (C ), 405 TSHttpSsnClientFdGet (C ), 420 TSError (C ), 405 TSHttpSsnDebugGet (C ), 405 TSEvent (C ), 381, 459 TSHttpSsnDebugSet (C ), 405 TSEventFunc (C ), 459 TSHttpSsnHookAdd (C ), 413 TSfclose (C ), 473 TSHttpSsnIsInternal (C ), 425 TSFetchWakeUpOptions (C ), 384 TSHttpSsnReenable (C ), 420 TSfﬂush (C ), 473 TSHttpStatus (C ), 385, 460 TSfgets (C ), 473 TSHttpTxn (C ), 460 TSFile (C ), 459 TSHttpTxnCachedReqGet (C ), 421 TSfopen (C ), 473 TSHttpTxnCachedRespGet (C ), 421 TSfread (C ), 474 TSHttpTxnCacheLookupStatusGet (C ), 420 TSfree (C ), 475 TSHttpTxnCacheLookupUrlGet (C ), 285, 420 TSfwrite (C ), 474 TSHttpTxnCacheLookupUrlSet (C ), 285, 421 TSHandleMLocRelease (C ), 438 TSHttpTxnClientFdGet (C ), 421 TSHostLookup (C ), 407 TSHttpTxnClientPacketDscpSet (C ), 422 TSHostLookupResult (C ), 459 TSHttpTxnClientPacketMarkSet (C ), 422 TSHostLookupResultAddrGet (C ), 407 TSHttpTxnClientPacketTosSet (C ), 422 TSHRTime (C ), 459 TSHttpTxnClientReqGet (C ), 423 TSHttpConnect (C ), 407 TSHttpTxnClientRespGet (C ), 423 TSHttpConnectWithPluginId (C ), 407 TSHttpTxnConﬁgFind (C ), 416 TSHttpEventNameLookup (C ), 406 TSHttpTxnConﬁgFloatGet (C ), 415 TSHttpHdrClone (C ), 408 TSHttpTxnConﬁgFloatSet (C ), 415 TSHttpHdrCopy (C ), 409 TSHttpTxnConﬁgIntGet (C ), 415 TSHttpHdrCreate (C ), 409 TSHttpTxnConﬁgIntSet (C ), 415 TSHttpHdrDestroy (C ), 409 TSHttpTxnConﬁgStringGet (C ), 416
525 Apache-Trafﬁcserver-Server Documentation, latest
TSHttpTxnConﬁgStringSet (C ), 416 TSMgmtFloat (C ), 460 TSHttpTxnDebugGet (C ), 405 TSMgmtFloatGet (C ), 439 TSHttpTxnDebugSet (C ), 405 TSMgmtInt (C ), 460 TSHttpTxnErrorBodySet (C ), 423 TSMgmtIntGet (C ), 439 TSHttpTxnHookAdd (C ), 413 TSMgmtString (C ), 460 TSHttpTxnIncomingAddrGet (C ), 424 TSMgmtStringGet (C ), 439 TSHttpTxnInfoIntGet (C ), 424 TSMgmtUpdateRegister (C ), 440 TSHttpTxnIntercept (C ), 425 TSMilestonesType (C ), 426 TSHttpTxnIsInternal (C ), 425 TSMiletonesType (C ), 389 TSHttpTxnMilestoneGet (C ), 426 TSMimeHdrClone (C ), 440 TSHttpTxnNextHopAddrGet (C ), 428 TSMimeHdrCopy (C ), 440 TSHttpTxnOutgoingAddrGet (C ), 428 TSMimeHdrCreate (C ), 441 TSHttpTxnParentProxySet (C ), 429 TSMimeHdrDestroy (C ), 441 TSHttpTxnReenable (C ), 429 TSMimeHdrFieldAppend (C ), 441 TSHttpTxnReqCacheableSet (C ), 285 TSMimeHdrFieldClone (C ), 441 TSHttpTxnServerAddrGet (C ), 429, 430 TSMimeHdrFieldCopy (C ), 442 TSHttpTxnServerFdGet (C ), 430 TSMimeHdrFieldCopyValues (C ), 442 TSHttpTxnServerIntercept (C ), 430 TSMimeHdrFieldCreate (C ), 442 TSHttpTxnServerPacketDscpSet (C ), 431 TSMimeHdrFieldDestroy (C ), 442 TSHttpTxnServerPacketMarkSet (C ), 432 TSMimeHdrFieldFind (C ), 443 TSHttpTxnServerPacketTosSet (C ), 432 TSMimeHdrFieldGet (C ), 443 TSHttpTxnServerReqGet (C ), 432 TSMimeHdrFieldLengthGet (C ), 443 TSHttpTxnServerRespGet (C ), 433 TSMimeHdrFieldNameGet (C ), 444 TSHttpTxnSsnGet (C ), 433 TSMimeHdrFieldNameSet (C ), 444 TSHttpTxnTransformedRespCache (C ), 433 TSMimeHdrFieldNext (C ), 444 TSHttpTxnTransformRespGet (C ), 433 TSMimeHdrFieldNextDup (C ), 444 TSHttpTxnUntransformedRespCache (C ), 433 TSMimeHdrFieldRemove (C ), 444 TSHttpType (C ), 387, 460 TSMimeHdrFieldsClear (C ), 449 TSInstallDirGet (C ), 435 TSMimeHdrFieldsCount (C ), 449 TSIOBuffer (C ), 460 TSMimeHdrFieldValueAppend (C ), 445 TSIOBufferBlock (C ), 460 TSMimeHdrFieldValueDateGet (C ), 446 TSIOBufferBlockReadStart (C ), 434 TSMimeHdrFieldValueDateInsert (C ), 445 TSIOBufferCopy (C ), 434 TSMimeHdrFieldValueDateSet (C ), 445 TSIOBufferCreate (C ), 434 TSMimeHdrFieldValueInt64Get (C ), 446 TSIOBufferDestroy (C ), 434 TSMimeHdrFieldValueIntGet (C ), 446 TSIOBufferProduce (C ), 434 TSMimeHdrFieldValueIntSet (C ), 446 TSIOBufferReader (C ), 460 TSMimeHdrFieldValuesClear (C ), 448 TSIOBufferSizedCreate (C ), 434 TSMimeHdrFieldValuesCount (C ), 448 TSIOBufferSizeIndex (C ), 460 TSMimeHdrFieldValueStringGet (C ), 446 TSIOBuffersSizeIndex (C ), 387 TSMimeHdrFieldValueStringInsert (C ), 447 TSIOBufferWaterMarkGet (C ), 434 TSMimeHdrFieldValueStringSet (C ), 448 TSIOBufferWaterMarkSet (C ), 434 TSMimeHdrFieldValueUintGet (C ), 446 TSIOBufferWrite (C ), 434 TSMimeHdrFieldValueUintInsert (C ), 448 TSIsDebugTagSet (C ), 405 TSMimeHdrFieldValueUintSet (C ), 448 TSLifecycleHookAdd (C ), 436 TSMimeHdrLengthGet (C ), 449 TSLifeCycleHookID (C ), 388 TSMimeHdrParse (C ), 449 TSLifecycleHookID (C ), 460 TSMimeHdrPrint (C ), 450 TSLookingUpType (C ), 389 TSMimeParser (C ), 460 TSmalloc (C ), 474 TSMimeParserClear (C ), 450 TSMBuffer (C ), 460 TSMimeParserCreate (C ), 450 TSMBufferCreate (C ), 438 TSMimeParserDestroy (C ), 451 TSMBufferDestroy (C ), 438 TSMLoc (C ), 460 TSMgmtCounter (C ), 460 TSMutex (C ), 460 TSMgmtCounterGet (C ), 439 TSMutexCreate (C ), 451
526 Apache-Trafﬁcserver-Server Documentation, latest
TSMutexDestroy (C ), 451 TSTextLogObjectRollingIntervalSecSet (C ), 456 TSMutexLock (C ), 451 TSTextLogObjectRollingOffsetHrSet (C ), 456 TSMutexLockTry (C ), 451 TSTextLogObjectRollingSizeMbSet (C ), 456 TSMutexUnlock (C ), 452 TSTextLogObjectWrite (C ), 456 TSNetAccept (C ), 452 TSThread (C ), 461 TSNetAcceptNamedProtocol (C ), 452 TSThreadCreate (C ), 456 TSNetConnect (C ), 453 TSThreadDestroy (C ), 456 TSOverridableConﬁgKey (C ), 390, 415 TSThreadFunc (C ), 461 TSParseResult (C ), 393, 460 TSThreadInit (C ), 457 TSPluginDirGet (C ), 435 TSThreadPool (C ), 399, 461 TSPluginInit (C ), 453 TSThreadSelf (C ), 457 TSPluginRegister (C ), 453 TSTrafﬁcServerVersionGet (C ), 457 TSPluginRegistrationInfo (C ), 460 TSTrafﬁcServerVersionGetMajor (C ), 457 TSrealloc (C ), 474 TSTrafﬁcServerVersionGetMinor (C ), 457 TSRecordAccessType (C ), 393 TSTrafﬁcServerVersionGetPatch (C ), 457 TSRecordCheckType (C ), 393 TSTransformCreate (C ), 459 TSRecordDataType (C ), 394, 461 TSTransformOutputVConnGet (C ), 459 TSRecordModeType (C ), 394 TSUrlClone (C ), 461 TSRecordPersistType (C ), 395 TSUrlCopy (C ), 461 TSRecordType (C ), 395 TSUrlCreate (C ), 461 TSRecordUpdateType (C ), 396 TSUrlDestroy (C ), 462 TSReleaseAssert (C ), 406 TSUrlFtpTypeGet (C ), 463 TSRemapDeleteInstance (C ), 454 TSUrlFtpTypeSet (C ), 463 TSRemapDone (C ), 454 TSUrlHostGet (C ), 463 TSRemapDoRemap (C ), 454 TSUrlHostSet (C ), 464 TSRemapInit (C ), 454 TSUrlHttpFragmentGet (C ), 463 TSRemapInterface (C ), 461 TSUrlHttpFragmentSet (C ), 464 TSRemapNewInstance (C ), 454 TSUrlHttpParamsGet (C ), 463 TSRemapOSResponse (C ), 454 TSUrlHttpParamsSet (C ), 464 TSRemapRequestInfo (C ), 461 TSUrlHttpQueryGet (C ), 463 TSRemapStatus (C ), 461 TSUrlHttpQuerySet (C ), 464 TSReturnCode (C ), 396, 461 TSUrlLengthGet (C ), 466 TSSDKVersion (C ), 396, 461 TSUrlParse (C ), 461 TSServerSessionSharingMatchType (C ), 397 TSUrlPasswordGet (C ), 463 TSServerSessionSharingPoolType (C ), 397 TSUrlPasswordSet (C ), 464 TSServerState (C ), 398, 461 TSUrlPathGet (C ), 463 TSSslContext (C ), 455 TSUrlPathSet (C ), 464 TSSslContextFindByAddr (C ), 455 TSUrlPercentEncode (C ), 465 TSSslContextFindByName (C ), 455 TSUrlPortGet (C ), 463 TSSslVConnection (C ), 470 TSUrlPortSet (C ), 464 TSSslVConnOp (C ), 398 TSUrlPrint (C ), 466 TSSslVConnReenable (C ), 469 TSUrlSchemeGet (C ), 463 TSstrdup (C ), 474 TSUrlSchemeSet (C ), 464 TSStringPercentDecode (C ), 465 TSUrlStringGet (C ), 466 TSStringPercentEncode (C ), 465 TSUrlUserGet (C ), 463 TSstrlcat (C ), 474 TSUrlUserSet (C ), 464 TSstrlcpy (C ), 474 TSVConn (C ), 461 TSstrndup (C ), 474 TSVConnAbort (C ), 467 TSTextLogObject (C ), 461 TSVConnCacheObjectSizeGet (C ), 467 TSTextLogObjectCreate (C ), 456 TSVConnClose (C ), 467 TSTextLogObjectDestroy (C ), 456 TSVConnClosedGet (C ), 467 TSTextLogObjectFlush (C ), 456 TSVConnCloseFlags (C ), 399 TSTextLogObjectHeaderSet (C ), 456 TSVConnFdCreate (C ), 467 TSTextLogObjectRollingEnabledSet (C ), 456 TSVConnIsSsl (C ), 468
527 Apache-Trafﬁcserver-Server Documentation, latest
TSVConnRead (C ), 468 TSVConnReadVIOGet (C ), 468 TSVConnShutdown (C ), 469 TSVConnSslConnectionGet (C ), 469 TSVConnTunnel (C ), 470 TSVConnWrite (C ), 470 TSVConnWriteVIOGet (C ), 470 TSVIO (C ), 461 TSVIOBufferGet (C ), 470 TSVIOContGet (C ), 471 TSVIOMutexGet (C ), 471 TSVIONBytesGet (C ), 471 TSVIONBytesSet (C ), 471 TSVIONDoneGet (C ), 471 TSVIONDoneSet (C ), 472 TSVIONTodoGet (C ), 472 TSVIOReaderGet (C ), 472 TSVIOReenable (C ), 472 TSVIOVConnGet (C ), 472 V Vol (C++ ), 284 Vol::evac_range (C++ ), 284 Vol::len (C++ ), 284 Vol::Vol::buckets (C++ ), 284 Vol::Vol::evacuate (C++ ), 284 Vol::Vol::segments (C++ ), 284 VolHeaderFooter (C++ ), 285 W write cursor, 516
528