Appinternals SAAS Configurationandreference Version 11.4.2

Configuring Aternity APM

The following sections explain configuration tasks to set up APM and customize its behavior to suit your application environment. Most configuration is done through the APM web interface. Some tasks require access to the agent systems. Major topics are as follows:

 “Web Interface Configuration“

 “Agent System Configuration and Metrics“

Aternity APM Version 11.4.2 1 2 Aternity APM Version 11.4.2 Web Interface Configuration

This material describes configuration screens in the APM analysis server web interface. See the “Agent System Configuration and Metrics“ topics for details on configuration tasks you perform on agent systems.

Aternity APM Version 11.4.2 1 2 Aternity APM Version 11.4.2 CHAPTER 1 SteelCental SaaS - Account

The ACCOUNT menu option opens a separate browser window with options to manage users. For SteelCentral SaaS, each company account corresponds to a separate analysis server in the cloud. The company name is displayed as the Realm value in the main screens of the APM interface. The Account screen lists the users that have been created for a company’s account. As described “Getting Started with Aternity APM SaaS“, when you first sign up for an APM SaaS account, the analysis server corresponding to a company’s account will have a single user with the “owner“ role. Users with the owner role can use this screen to add new users or remove and modify existing users. See “User Roles“ for details on the different roles. The main analysis server interface shows the company account name and the email address for the user that is logged in:

Account Details

The Account Details area shows the company name and company ID along with the user email address and name. The company name is displayed as the Realm value in the main screens of the APM interface. The company ID is used to specify the analysis server when installing agent software on monitored systems.

Edit Profile and Change Password Options

In the Account Details area, any user can change the settings that apply to their own profile:

 Click Edit Profile to open the Edit User screen. You can change the value of the Name field. This is a display name for the user, but is currently used only in the Account screen. Click Edit User to save the change. Use the browser back button to navigate back to the Account screen.

Aternity APM Version 11.4.2 3 SteelCental SaaS - Account

 Click Change Password to open the Change Password screen. Supply the old password, new password and confirmation, and click Change Password. Use the browser back button to navigate back to the Account screen. This option is not available if the user was configured to use SAML single sign-on access, as indicated by saml in the Type column in the list of users (see “Adding Users: SAML-Enabled Companies“ for details).

SAML Configuration Option

The SAML Configuration option is available only for users with the “owner“ role, and only when this feature is enabled for the company (also called realm) by Aternity APM SaaS operations. Click SAML Configuration to open the “SAML Configuration“ screen.

Adding Users

Only users with the “owner“ role can add users. You must supply a valid email address for each new user added. Users are identified through the address, and adding a new user triggers invitation and validation email messages that the user must interact with. To add a user, use follow these steps:

1) Click the ACCOUNT option in the menu to open the Account screen.

2) Click Add Users at the top of the Users list.

3) The Add User screen opens. Supply the user’s email address and choose a role:

4 Aternity APM Version 11.4.2 SteelCental SaaS - Account

4) Click Add User and use the browser’s back button to reopen the Account screen. Notice that the user you added is present, with a status of Invited...:

5) Adding a user triggers an invitation email to the address you specified. The new user clicks the Join Now link in the email:

6) Clicking the Join Now link opens a web page where the user supplies a display user name (currently used only in the Account screen) and password. The new user clicks the Join button:

Aternity APM Version 11.4.2 5 SteelCental SaaS - Account

7) Clicking Join triggers another email to the new user. This email contains a Verify Now link. The new user clicks the link to open a web page that verifies the email address and links to the analysis server sign-in page:

Adding Users: SAML-Enabled Companies

If SAML-based authentication has been enabled in the “SAML Configuration“ screen, the Add User screen has an additional User Type setting:

 Choose Local for the user to authenticate with the APM SaaS authentication server.

 Choose SAML for the user to authenticate with the IdP server configured in the SAML Configuration screen. You must choose SAML to take advantage of single-sign on authentication. Note—If you choose SAML, the email address must be the full canonical address, not an alias.

Modifying Other Users

Only users with the owner role can modify other users.

 Click the edit icon ( ) to change the user’s role (see “User Roles“).

 Click the delete icon ( ) to delete the user.

User Roles

The user account you use to log in to the APM interface has different capabilities depending on the roles it has been assigned.

6 Aternity APM Version 11.4.2 SteelCental SaaS - Account

The following table describes the roles and their capabilities.

Role Description

read-only Log in and access “Data Analysis Tabs and Metric Data Views“ in APM. Accounts with the read-only role cannot perform configuration tasks. The CONFIGURE menu and its options are not available:

Accounts with the read-only role can click the ACCOUNT menu option, edit their own profile, and see the names of users with the owner role. Because accounts with the read-only role cannot perform configuration tasks, the following links are not available: • The link to the “Define a Transaction Type Screen“ from an empty “Top Transaction Types Card“is not available • Direct URL links (for example, if someone copies/pastes a URL for a configure screen) will be redirected to the Overview tab. However, note that the “Alert History Screen“ (an option under the CONFIGURE menu) is still accessible (via the “Recent Alerts Card“ on the Overview tab). admin Accounts with the admin role have read-only access plus access to the CONFIGURE menu and its options. owner Accounts with the owner role have admin access plus the ability to: • Add and modify other users • Configure SAML single sign-on access (this configuration is available only when this feature is enabled for the company (also called realm) by Aternity APM SaaS operations)

Aternity APM Version 11.4.2 7 SteelCental SaaS - Account

8 Aternity APM Version 11.4.2 CHAPTER 2 Install Agents Screen

Note: This Screen Not Available to Read-Only Users Configuration screens are available only when you log in to APM using an administrative account. See the “User Roles“ topic for details on administrative and read-only accounts.

This screen provides an easy way to download agent software to your computer. It is an alternative to downloading the agent software from the APM support page here: https://support.riverbed.com/content/support/software/opnet-performance/appinternals-xpert.html. The links on this page provide access to more recent agent versions that are compatible with this release of the analysis server:

 Long-Term Support (LTS) agent releases are maintenance releases that contain important problem fixes only. These versions correspond to the latest release available on the support page.

 Latest agent releases contain new features and fixes to problems found in versions since the stable release branch.

Agent Installation Steps Step : Click the link for the software for the operating system on which you want to install the agent. For more details on agent installation, see the following installation topics:

 “Agent Installation: Windows“

 “Agent Installation: Unix-Like Operating Systems“ Step (SaaS): Agents that connect to a SaaS analysis server require a customer ID to connect. When you install such agents, choose the SaaS option and the installation prompts for the customer ID. Supply the Customer Id value shown here. Step (on-premises): For agents that will connect to an analysis server installed in your environment (“on premises”), the agent installation prompts for the location of the analysis server. Use the Analysis Server hostname value shown here.

Recently Discovered Agents After you install and start the agent, it should connect to the analysis server and appear in the Recently Discovered Agents list. Click the View all Agents link below the list to open the “Agent List Screen“.

Aternity APM Version 11.4.2 9 Install Agents Screen

10 Aternity APM Version 11.4.2 CHAPTER 3 Agent List Screen

This screen lists agent systems being monitored by the APM analysis server. Each row in the display shows details about an agent. Click on column headings to sort rows in ascending order based on the column value. Click the column heading again to sort in descending order. An icon indicates ascending or descending sort order. The Agent column shows the icon representing the “Agent Status“ and the agent name. Sorting by ascending Name sorts as follows:

 Fully-qualified-domain names are grouped together by matching domain name components. For example, the agent name myagent.lab.riverbed.com will be grouped with all other agents that share the lab.riverbed.com domain. Similarly, myagent.riverbed.com will be grouped with all other agents that share the riverbed.com domain.

 Within groups of matching domain names, agent names are sorted alphabetically.

 Names without a domain are grouped together.

 Agent names that end in numbers are grouped together and sorted numerically.

Controls and Operations

Limiting Agents Displayed: Filter Criteria in Left Pane

Limit the agents listed by clicking one or more of the criteria to the left:

 Each criterion shows the number of matching agents in parentheses. Click a criterion to limit the displayed agents to those that match it. You can click additional criteria to further filter the list.

 Entries under tags show all the “Tags“ keys defined on agents. Click a key value to expand that tag and display different values for that key. Click a key or value to filter on that component.

Aternity APM Version 11.4.2 11 Agent List Screen

Limiting Agents Displayed: “Filter by agent name or tag...” Text Box

Limit the agents listed by typing a search string in the text box. The search matches agent names or “Tags“ that contain the string after you press ENTER. The analysis server matches strings as follows:

 If the string does NOT contain an equals-sign character (=), it matches any of the following values that contain it: – agent name –tag key name –tag value

 If the string DOES contain an equals-sign character (=) it matches tag names and values only. (Strings with an equals-sign character will not match agent names.) The matching works as follows: – The sub-string to the left of the first equals-sign character (=) must be an exact match for a tag key. – The sub-string to the right of the first equals-sign character (=) matches any tag value that contains it. For example, consider the following tag key-value pair : docker hostname = nhv1-ubnt15-1 The search string docker = ubnt will not match the tag because it is not an exact match for the docker hostname key. The following search strings will match the tag: docker hostname = docker hostname= docker hostname=ubnt docker hostname = ubnt ubnt docker

 Search strings are NOT case sensitive.

 Wildcards are not allowed.

“show / hide offline agents that released licenses”

This option appears at the bottom of the screen when both the following conditions are true:

 The analysis server is monitoring agents that are configured to release license units when they are offline.

 Any of those agents are “Offline“. By default, such agents do not appear in the agent list. In cloud environments, agent virtual machines can be short-lived and release licenses when they terminate. There may be many such agents and it is typically not useful to include them in the list. Click show offline agents that released licenses to display such agents. The option changes to hide offline agents that released licenses. Click it to again hide them. After an agent has been offline for more than 30 days, the analysis server removes its entry altogether and it will not appear even when you click show offline agents that released licenses.

Associating WMI Custom Metrics with a Windows Agent

You can enable multiple WMI Custom Metric configurations for a Windows agent from the Agent List screen. For more information, see “Enabling Custom Metrics Configurations for WMI“.

12 Aternity APM Version 11.4.2 Agent List Screen

Agent Status

Each agent has a status icon that represents its status. These status icons also appear in the “Agent Details Screen“ and “Process List Screen“.

Status Meaning

New An agent appears as new for 7 days or until one or more processes to instrument are successfully instrumented. (You instrument processes to instrument in the “Agent Details Screen“.) Good Health Agent reports a healthy status for all its components.

Degraded Agent reports at least one component as non-healthy. If the status is Degraded, pausing the mouse pointer over the agent shows detail that includes the individual agent components that are non-healthy and their status. Critical Agent reports at least one component as not functioning. If the status is Critical, pausing the mouse pointer over the agent shows detail that includes the individual agent components that are non-healthy and their status. Connected but Agent is online but the analysis server received an error retrieving status. An agent is unable to get considered online as long as the analysis server’s WebSocket connection to it is open. For status from agent example, the analysis server reports this status if an agent system crashes suddenly and the agent did not stop its Windows services. If the agent system remains down, this status will change to Offline. Offline The analysis server cannot connect to the agent. This may be because the agent software is not running, the agent system is down, the network is unavailable, or some other cause.

Where to Go from Here

For more information on working with agents, see the following topics:

 “Retain / Release Licenses on Disconnect“ option in the “Agent Details Screen“

 ““show / hide processes of offline agents that released licenses”“ option in the “Process List Screen“

 “Deploying Agents in Dynamic Environments“

 “Managing License Unit Usage“

Aternity APM Version 11.4.2 13 Agent List Screen

14 Aternity APM Version 11.4.2 CHAPTER 4 Agent Details Screen

This screen shows details about an APM agent and provides options that control monitoring of processes and other activity on the system where the agent is installed.

 The Agent Details screen allows you to enable a single WMI Custom Metrics configuration for a Windows agent, as well as a single JMX Custom metrics configuration for a single process. For more information, see “Enabling Custom Metrics Configurations“.

 The “Agent Information and Options“ area shows a summary of the agent status and includes options to manage “Tags“, perform “Actions“, and show “More Details and Options“.

 The “Process List and Options“ area lists discovered processes to instrument (see “What Are ‘Processes to Instrument’?“) with options to manage whether they are monitored.

 The “Processes to Instrument Details“ area appears below the process list when you select a process in the list.

 The “Network Data (SaaS Analysis Server Only)“ area control whether the agent collects network data, where to send that data, and optionally filter which traffic will be monitored.

 The “Process Data (SaaS Analysis Server Only)“ area has a setting that controls whether the analysis server reports data from the “Processes Sub-Agent“.

Aternity APM Version 11.4.2 15 Agent Details Screen

 The “JMX Data“ area has settings that control whether the analysis server reports data from the “JMX Sub-Agent“ for Java processes that are not already monitored by the “JIDA Sub-Agent“.

Note: Refresh the browser to see changes on agent This screen does not automatically update to reflect changes on the agent system, such as agent health, uptime, and status of processes to instrument. You must refresh the web page display to see changes. (Or you can navigate to another screen and back.)

16 Aternity APM Version 11.4.2 Agent Details Screen

“Harvesting Disabled” Banner

This banner appears at the top of the screen if any processes to instrument have been configured so the analysis server will not retrieve or display transaction trace data. It appears when any process has the following combination of options set in the “Edit Processes to Instrument Dialog“:

 The “Instrument Option (Restart Required)“ enabled

 The same process has the “Harvest Option“ disabled. The banner is a reminder that, for those processes, the analysis server will not retrieve or display transaction trace data. Scan the Harvest column in the “List of Processes to Instrument and Assign JMX and WMI Custom Metric Configurations“ to see which processes have harvesting disabled.

Agent Information and Options

This area shows details and configuration options for the agent. The agent name and its status (see “Agent Status“) appear at the top of the area, along with a summary of the agent:

Aternity APM Version 11.4.2 17 Agent Details Screen

 Hostname shows the system name as returned by the hostname command and the “Agent Status“ . The Hostname value is a link. Click the link to open the “Servers Tab“ filtered by the hostname value.

 Components shows the overall “Agent Status“ for agent components. If the status is Critical or Degraded, the status shows the individual agent components that are non-healthy and their status.

 Licenses Consumed shows the number of license units in use by instrumented processes on the agent and the current setting for the “Retain / Release Licenses on Disconnect“ option in the “Actions“ list. In addition to a summary of the agent status, this area includes options to manage “Tags“, perform “Actions“, and show “More Details and Options“.

Tags

Tags are custom identifiers that categorize servers that APM is monitoring. Create tags to organize and categorize servers in any way that suits your environment. Tags are especially useful in dynamic environments, where servers are cloned many times and are created and destroyed frequently. In these environments, tags identify servers without relying on static host names or IP addresses. In environments where IP addresses and host names are transient, tags provide an easy way to see performance data for specific systems that you are interested in.

Note: Container Tags: Automatically-created tags for Docker containers As described in “Docker Container Tags“, container tags are a special type of tags. They are automatically generated by the agent and provide details about individual Docker containers. tags are key-value pairs. For example, you could create the tag key-value pair Role - WebServer on all agents that are monitoring web servers. In the “Servers Tab“, you can right-click on the server name to filter on that tag and limit results to those systems, even if they changed host names many times:

18 Aternity APM Version 11.4.2 Agent Details Screen

Note: Restart NOT Required You do not need to restart the agent or (for Version 10.9 agents and later) applications for new or changed tags to take effect. However, remember that in general it takes several minutes for data reflecting changes to propagate from agents to the analysis server interface.

Where Tags Appear in the APM Interface

The APM interface exposes tags as follows:

 As described in “Using the Special “logical server” Tag to Condense Application Maps“, in the “Application Map Tab“, when users choose the “Logical Servers“ Edit option.

 In the “Servers Tab“ table: – As a right-click filter option of the “Server“ column. This option shows the tag names and values. Click a tag to add a global filter to the “Filter Area“). –In the “Tags“ table column

 In the “Search Tab“: –In the “Tags“ column of the “Search Results“ table –As the “tag“, “tag.name“, and “tag.value“ search fields

 In the “Tags“ section of the “Transaction Details“ window

 In the “Limiting Agents Displayed: Filter Criteria in Left Pane“ shown in the left of the “Agent List Screen“

 In the “Instances Tab“.

 In the “Application Map Tab“.

 In the “Environmental Thresholds“ area on the “Threshold Overview Screen“.

 In the “Add Tag / List of Tags“ in this screen

Add Tag / List of Tags

This area lists any tags that have been defined for this agent. If the agent version is AppInternals Version 10.9 or later, you can add or edit tags:

 To add a new tag, click Add Tag. The “Add/Edit Tag Dialog“ opens.

 To edit a tag, click the edit icon ( ) in the row for the tag. The “Add/Edit Tag Dialog“ opens.

 To delete a tag, click the delete icon ( ) in the row for the tag. (You cannot delete the special logical tag. See “Using the Special “logical server” Tag to Condense Application Maps“ for details on this tag.)

Aternity APM Version 11.4.2 19 Agent Details Screen

 The tags values are links. Click a link to open the “Servers Tab“ filtered by the tag value.

Note: Add, Edit Options Not Available for Older Agents If the agent version is earlier than AppInternals Version 10.9, create tags by editing a file on the agent system. See “Creating tags with the tags.yaml File for Agents Running in PaaS Environments“ in the “Deploying Agents in Dynamic Environments“ section. The list here updates to reflect tags created or modified via the tags.yaml file.

Add/Edit Tag Dialog

Use this dialog to add or edit the key and value for tags. Note the following rules for tag keys and values:

 Keys and values can contain letters, numbers, and spaces. Other characters are not allowed.

 Specify multiple values for a key as a a comma-separated list. For example: value 1, value 2, value 3 .

 Tag values are optional.

 Keys are case sensitive. Tag1 : foo and tag1 : foo results in separate tags.

 You cannot modify the special logical server key, only its value. See “Using the Special “logical server” Tag to Condense Application Maps“ for details on this tag.)

Using the Special “logical server” Tag to Condense Application Maps

The logical server tag always appears at the top of the list of tags. Use the value of this special tag to control the number and names of servers in the application map in the “Application Map Tab“ and the “Application Map Card“. The application map uses the value of the logical server tag key to determine the number and names of “Server Nodes“ in the application map when users choose the “Logical Servers“ Edit option. In environments with dynamically-created servers, or many servers, use logical tags to reduce clutter and supply custom server names in the application map:

 All agents with the same value of the logical server tag key will be shown as a single server that uses the value as the server name.

 Agents that do not have a logical server tag key will not appear in the map.

20 Aternity APM Version 11.4.2 Agent Details Screen

As a simple example, consider an environment with four agents. The application map “Physical“ Edit option shows the four agents:

You can simplify the map to show two servers named Tier 1 and Tier 2.

1) For two of the agents, click the edit icon ( ) for the logical tag key and supply a value of Tier 1.

2) On the other two agents, supply a value of Tier 2.

3) In the application map, choose the “Logical Servers“ Edit option. The application map shows only two servers with the names you specified:

Aternity APM Version 11.4.2 21 Agent Details Screen

Keep in mind the following behavior about the logical server key:

 An empty value for the logical server key will have no effect on the application map.

 Multiple values for the logical server key will have no effect on the application map.

 If you change the value of the logical server key, the application map uses the most recent value during the time interval specified in the “Time Range Settings“ area.

 If you specify the logical server key in tags.yaml file, and there are multiple occurrences of the key, the application map uses last value in the file.

More Details and Options

Click the More details and options link to see more details and options for the agent.

Additional details include:

 Operating system version

 Status of the agent connection with the analysis server

 Available disk space for performance data In addition to this detail, expanding the Show Support Information area exposes options to create a “diagnostic bundle” (“Download the agent logs and diagnostic files“)and remove offline agents from the analysis server (“Delete“).

Download the agent logs and diagnostic files

A “diagnostic bundle” is a compressed package of analysis server log files and output from commands and troubleshooting scripts. Typically, you generate a diagnostic bundle at the request of Aternity support. Click this link to generate a diagnostic bundle for the agent and open the “Diagnostic Bundle Screen“, where you can download it to your local system.

22 Aternity APM Version 11.4.2 Agent Details Screen

Delete

The Delete button is enabled only if the agent status is Offline. Click Delete to remove the agent from the “Agent List Screen“ and open that screen. Deleting an agent does not affect it or any data it may have generated. If the agent restarts and contacts the analysis server again, it will reappear in the agent list. Deleting an agent is useful if you know the agent will not restart, such as when you no longer want to monitor a system and have removed the agent software from that system. Deleting the agent removes the obsolete entry from the agent list and other configuration screens. Deleting an agent can also be useful even when you expect it to restart. It forces the agent to rediscover all candidate processes to instrument and send a new list to the analysis server. This will remove any incorrect and obsolete entries from the “List of Processes to Instrument and Assign JMX and WMI Custom Metric Configurations“. (However, you will need to re-select the processes you want to instrument and re-associate them with custom configuration settings.) Follow these steps:

1) On the agent system, shut down the agent (On Windows systems, stop the Riverbed SteelCentral AppInternals Agent service. For other operating systems, see “Controlling Agent Software on Unix-Like Systems“).

2) On the agent system, Delete the jobs.json file in the \Panorama\hedzup\mn\userdata directory.

3) On the this screen, click Delete.

4) On the agent system, restart the agent.

Actions

Click the Actions button to show options that affect all processes that the agent discovered, or the agent itself (only the “Retain / Release Licenses on Disconnect“ option).

Instrument All

Sets all processes to load the APM monitoring library when they next restart. This is the same as selecting the Instrument option in the “Edit Processes to Instrument Dialog“ for every process. See the “Instrument Option (Restart Required)“ for more details.

Stop Instrumenting All

Sets all processes to unload the APM monitoring library when they next restart. This is the same as clearing the Instrument option in the “Edit Processes to Instrument Dialog“ for every process. See the “Instrument Option (Restart Required)“ for more details.

Start Harvesting All

Sets the analysis server to retrieve transaction trace files for all processes. This is the same as selecting the Harvest option in the “Edit Processes to Instrument Dialog“ for every process. It will only apply to processes that are already being instrumented . This option may use additional license units. See the “Harvest Option“ for more details.

Aternity APM Version 11.4.2 23 Agent Details Screen

Stop Harvesting All

Sets the analysis server to stop retrieving transaction trace files for all processes. This is the same as clearing the Harvest option in the “Edit Processes to Instrument Dialog“ for every process. This option may free up license units. See the “Harvest Option“ for more details.

Retain / Release Licenses on Disconnect

These options control whether the analysis server will release any license units used by an agent when it cannot connect to that agent (in other words, when the agent status is “Offline“). Releasing license units is typically useful in cloud environments, where agent virtual machines can be short-lived and re-initialize frequently as a different virtual machine under a different host name. Depending on the current setting (shown in the Licenses Consumed detail in “Agent Information and Options“), the option will be either Releases Licenses on Disconnect or Retains Licenses on Disconnect. Click the option to change the setting. By default, APM agents earlier than Version 10.3 retain licenses when they are offline. Version 10.3 and later agents release licenses when they are offline. See these related topics:

 “Managing License Unit Usage“

 ““show / hide offline agents that released licenses”“ option in the “Agent List Screen“

Process List and Options

This area displays the “List of Processes to Instrument and Assign JMX and WMI Custom Metric Configurations“ and Assign JMXand WMI configurations. Each row in the list shows settings and status for the processes. Click a row in the list to show details in the “Processes to Instrument Details“ below the list.

Note: Use the Process List Screen for Bulk Configuration of Processes Similar to the “List of Processes to Instrument and Assign JMX and WMI Custom Metric Configurations“ on this screen, the “Process List Screen“ shows processes to instrument, but for all agents. You can change settings for multiple processes at once on that screen, something you cannot do on this screen.

What Are ‘Processes to Instrument’?

One of the functions of the APM agents is to detect applications that are candidates for monitoring. The agents discover applications at the “processes to instrument” level, which corresponds to one or more processes running on the system. APM instruments and monitors processes to instrument as a single unit:

 Configuration options in the “Define a Configuration Screen“ apply at the processes to instrument level.

 Data is reported in the web interface at the processes to instrument level.

24 Aternity APM Version 11.4.2 Agent Details Screen

 Each row in the “Processes to Instrument“ list represents processes that the agent detected were running on the agent system: – Click the edit icon ( ) in a row to open the “Edit Processes to Instrument Dialog“ and change whether APM will monitor the processes. – Select a row to see detailed status in the “Processes to Instrument Details“ area in the lower right of the screen. – Columns show some of the current settings for the processes and link to additional settings.

All | Host | Docker: Filter Processes for Docker Hosts

These options appear only if the agent is a Docker host. (See “Monitoring Docker Containers“ for details for configuring the agent in Docker environments.) Use the options to filter the “List of Processes to Instrument and Assign JMX and WMI Custom Metric Configurations“ list as follows:

 All: show all detected processes (this is the default)

 Host: show only processes detected as running on the Docker host (in other words, processes that are not running in a container)

 Docker: show only processes that are running in a Docker container. When you choose this option, the process list adds the Container ID column that shows the short Docker container ID.

List of Processes to Instrument and Assign JMX and WMI Custom Metric Configurations

Note: Not Seeing Java Applications on Solaris? Be Sure to Enable Instrumentation and Restart! Applications running on the Solaris operating system require manual configuration before they will appear in this list. You must configure the application to load the APM profiler library and restart the Java process before it will appear in this list. Then, to monitor the application, select the “Instrument Option (Restart Required)“ and restart the application again. See “Enabling Java Instrumentation on Solaris“ for more details.

The list of processes is presented as a table. You can click on most column headings to sort the column in ascending order based on the column value. Click the column heading again to sort in descending order. An icon indicates ascending or descending sort order.

Aternity APM Version 11.4.2 25 Agent Details Screen

Processes to Instrument

This column shows the processes to instrument that the APM agent discovered on the system it is monitoring. Icons indicate if the process is Java ( ) or .NET (), and if it is in a Docker container (). If any process is in a Docker container, the “All | Host | Docker: Filter Processes for Docker Hosts“ options will be visible. Each name may correspond to multiple processes. This is common with Microsoft Internet Information Services (IIS) application pools, which often have multiple AppDomains running in them. It also occurs when there are multiple Java or applications running under the same name. The names that correspond to multiple processes are links with a dashed underscore. Click the link to open a dialog.

Modify AppDomains Dialog For .NET processes (including application pools), the link opens the Modify AppDomains dialog. APM automatically discovers new AppDomains as they start and displays them in this list as long as they are running. By default, all AppDomains will be instrumented if the “Instrument Option (Restart Required)“ for the process is selected. Clear the Include? check box to prevent specific AppDomains from being instrumented. Any changes you make here are reflected in the “AppDomain Blacklist/Whitelist“ settings in the “Define a Configuration Screen“ for the configuration to which the processes are assigned. The AppDomain Blacklist/Whitelist settings provides more control over the AppDomains that APM will instrument and monitor. The blacklist or whitelist affect the behavior of the Include? check box as follows:

 If an AppDomain matches an entry the blacklist for the configuration, its Include? check box is unchecked. (If the matching entry included a wildcard, the Include? setting is also disabled.) You must modify the blacklist to change whether the AppDomain is instrumented.

 Similarly, if an AppDomain matches an entry in the whitelist, its Include? check box is checked. (If the matching entry included a wildcard, the Include? setting is also disabled.)

View Processes Dialog For Java applications, the link opens the View Processes dialog that shows the processes and their process identifier. Unlike with .NET AppDomains, you cannot exclude specific Java processes from instrumentation. Once you select the “Instrument Option (Restart Required)“, the processes-to-instrument name appears in the analysis screens in many places, labeled as Instance. In the Search tab, you can search for a specific processes-to-instrument name using the “instance“ search field. You can change the name by selecting the row and clicking Rename in the “Processes to Instrument Details“ area. This changes the name throughout the APM interface.

Configuration

This column shows the name of the “configuration” with settings that control what application data that JIDA or the dotNet sub-agent collects for the processes-to-instrument name. Click the name to open the “Define a Configuration Screen“ for the applicable configuration. You can change the assigned configuration in the “Processes to Instrument Details“ area. (You can also change the assigned configuration in the “Process List Screen“.)

26 Aternity APM Version 11.4.2 Agent Details Screen

Instrument

Whether the process will load (or unload) the APM monitoring library when it next restarts. This column reflects the status of the “Instrument Option (Restart Required)“ in the “Edit Processes to Instrument Dialog“. Possible values are:

“Instrument” Value Meaning

Enabled Instrument option is selected (checked) and in effect.

Disabled Instrument option is cleared (unchecked) and not in effect.

Error Instrument option is selected but the agent reported an error for the instrumentation health of the process. To see more details, select the row and click the “Instrumentation Health“ View report link in the “Processes to Instrument Details“ area in the lower right of the screen. Failed Instrument option was changed, but the analysis server could not deliver the changed setting for the processes to instrument to the agent (for instance, because the agent is not running). Updating… Instrument option was changed and the analysis server is communicating the change to the agent. When the operation finishes, the Instrument value will update to show another value. Warning Instrument option is selected but the agent reported a warning for the instrumentation health of the process. To see more details, select the row and click the “Instrumentation Health“ View report link in the “Processes to Instrument Details“ area in the lower right of the screen.

Harvest

Whether the analysis server will retrieve transaction trace files for this process and whether the process (possibly) uses a license unit. This column reflects the status of the “Harvest Option“ in the “Edit Processes to Instrument Dialog“. If the “Instrument“ column value is Disabled, the Harvest value is always n/a (not applicable). This is because the Harvest option is disabled unless the Instrument option is selected.

Status

This status indicates whether the processes are instrumented (if known) and whether they must be restarted on the agent system. The status changes whenever you change the “Instrument Option (Restart Required)“ or change the corresponding configuration in the “Define a Configuration Screen“.

“Status” Value Meaning

Running The processes are running but not instrumented.

Not Running The processes are not running.

Aternity APM Version 11.4.2 27 Agent Details Screen

“Status” Value Meaning

Awaiting Restart The processes are running but require a restart for changes to take effect. Changing the “Instrument Option (Restart Required)“ always requires restarting the processes before the changes takes effect. •If Instrument is checked and the status is Awaiting Restart then the process is not currently instrumented. After restarting, the status updates to Instrumented. •If Instrument is not checked and the status is Awaiting Restart then the process is currently instrumented. After restarting, the status updates to Running. Note: Awaiting Restart may also mean that Java instrumentation is not enabled For Java processes, if you have not enabled instrumentation as described in “Enabling Instrumentation of Java Processes“, the instrumentation status will show as Awaiting Restart, but restarting the affected processes will have no effect until you enable Java instrumentation. May Require Restart The processes are running and instrumented, but may require a restart for certain changed settings to take effect. For more details, see “Settings That Require Restarting Processes“ in the “Define a Configuration Screen“ documentation. Instrumented The processes are running and instrumented.

Not Instrumentable This status appears only for Java processes to instrument and means that the Java version is not supported. Java 1.5 is the earliest supported version. Mixed This status applies only to processes to instrument that represent multiple processes. It means that individual processes have different statuses (such as Instrumented and Awaiting restart). Unknown The analysis server could not obtain the status of the processes to instrument (for instance, because the agent is not running).

Edit

Click the edit icon ( ) to open the “Edit Processes to Instrument Dialog“. This dialog box has settings to rename, change the assigned configuration, or change the monitoring state for the corresponding processes to instrument.

Delete (X)

Click the X icon to remove a processes-to-instrument name from list. This is useful for processes that:

 Are not running

 You know you do not to monitor

 Are not likely to run again on the agent system If the processes do restart, they will reappear in the list. Deleting processes does not remove any data they may have generated while instrumented.

28 Aternity APM Version 11.4.2 Agent Details Screen

Processes to Instrument Details

When you select a row in the “List of Processes to Instrument and Assign JMX and WMI Custom Metric Configurations“, details for the processes appear in the lower right of the screen. Some of the details are the same as columns in the “List of Processes to Instrument and Assign JMX and WMI Custom Metric Configurations“.

Name The same value as the “Processes to Instrument“ column in the list of processes to instrument. Discovered As The name under which the processes were discovered. Unless the processes were renamed in the “Edit Processes to Instrument Dialog“, this value will be the same as Name. Framework The application framework (Java, .NET).

Application Server The application server type (such as IIS or WebLogic) where applicable and where APM can determine it.

Instrument The same value as the “Instrument“ column in the list of processes to instrument.

Harvest The same value as the “Harvest“ column in the list of processes to instrument.

Configuration The same value as the “Configuration“ column in the list of processes to instrument.

Configuration The status of communication of configuration changes to the agent. When users change Status configuration settings for discovered processes, the analysis server attempts to send new configuration options to agents affected by the changes. This occurs for the following changes: • Changing the “Instrument Option (Restart Required)“ (either selecting or clearing the option) in the “Edit Processes to Instrument Dialog“ or “Process List Screen“ • Changing options in the “Define a Configuration Screen“ for the configuration to which the processes are assigned Possible Configuration Status values are: • Synchronized: The analysis server sent the configuration options to the agent and received acknowledgment. (The configuration status is also considered Synchronized when an agent discovers processes to instrument and they appear in the list. But the Synchronized status does not appear until the Instrument option is selected or cleared.) • Pending: The analysis server has not yet sent the configuration options to the agent. • Failed: The analysis server sent the configuration options to the agent but either did not receive an acknowledgment or received an error in response. This status is typically caused by the timing of communication of the analysis server with the agent. It usually changes to Offline or another status the next time the analysis server contacts the agent. • Offline: The analysis server could not connect to the agent. Instrumentation The same value as the “Status“ column in the list of processes to instrument. Status

Last Trace When the analysis server last retrieved a transaction trace file for the processes. Harvested

Aternity APM Version 11.4.2 29 Agent Details Screen

Licenses Whether the processes use a license unit. Only processes for which the “Harvest Option“ Consumed in the “Edit Processes to Instrument Dialog“ is selected use a license unit. Note that, for .NET processes, only one process on an agent will use a license unit. However, this message appears for every .NET process, even those that do not use a license unit. Instrumentation Whether the agent detected any problems for instrumenting the process. The agent Health checks for several conditions that may cause instrumentation problems and assigns a health level of Error or Warning if it detects any. Other possible values for Instrumentation Health are Good (no potential problems detected) and Not Instrumented (the Instrument option is not selected). Click the View report link to open a tabbed window with detailed messages. Scan for messages with a Level of Error or Warning.

Edit Processes to Instrument Dialog

This dialog opens when you click the “Edit“ icon ( ) in the “List of Processes to Instrument and Assign JMX and WMI Custom Metric Configurations“. Use settings here to rename processes, change their assigned configuration, or change their monitoring state.

Renaming Processes (Restart Required)

Change the processes-to-instrument name by clicking Rename. This changes the name throughout the APM interface. (The name appears in the analysis screens in many places, labeled as Instance.) Changing the name of the processes to instrument changes the “Status“ to May Require Restart, but always requires restarting the associated processes.

Renaming ASP.NET Web Applications May Have No Effect

Renaming ASP.NET web applications may not change the name displayed elsewhere in the APM interface. This includes changing the name of IIS-hosted application pools (such as IIS_DefaultAppPool).

30 Aternity APM Version 11.4.2 Agent Details Screen

When you rename an IIS application pool (worker process), you change the name (such as IIS_DefaultAppPool) in the Define a Configuration screen. You also change the instance name for the default AppDomain. However, the code run in the default AppDomain of an IIS worker process has only a single responsibility: create an appdomain for a web application to execute within. Almost all (if not all) of the code that does this is part of the .NET Framework (System.* classes, for example) and is not instrumented by default. Because of this, you typically will not see the renamed AppDomain instance for the default appdomain.

Changing Assigned Configuration

Change the configuration to which this processes-to-instrument name is assigned by changing the Name value and clicking Save. Changing the configuration takes effect immediately. The processes do not have to be restarted. Create a new configuration by clicking New to open the “Define a Configuration Screen“ with a new configuration. New configurations are automatically named config - x. You can rename them in “Current Configuration“ area of the Define a Configuration screen.

Changing Monitoring State

The Instrument, and Harvest options control the state of APM monitoring for the processes. The Harvest option may also affect whether processes use up a license unit.

Instrument Option (Restart Required)

This option controls whether the process will load (or unload) the APM monitoring library when it next restarts. This “instrumentation” is the mechanism that APM uses to monitor performance. Selecting or clearing the Instrument check box requires that the processes be restarted on the agent system. The “Status“ should change to “Awaiting Restart“.

Note: Before Instrumenting, Manually Enable Instrumentation for Java Processes on the Agent Before you select any Java processes to instrument, be sure that instrumentation has been enabled for them on the agent system. If instrumentation has not been enabled, APM will not monitor the processes. The “Status“ will show as Awaiting Restart, but restarting will have no effect. See “Enabling Instrumentation of Java Processes“ for details.

 Select the Instrument option to start monitoring the corresponding processes. When you select this option, the analysis server sends the agent detailed options specified by the configuration named in the Configuration column. (See “Define a Configuration Screen“ for details.)

Aternity APM Version 11.4.2 31 Agent Details Screen

 Clear the check box to stop monitoring the processes.

Note: Reinstalling Agent Causes Inconsistent State When processes are successfully instrumented, the Instrument option should be checked and the “Status“ should be “Instrumented“. In some cases, the Instrument option will be checked but the Status will show as Running. This is an inconsistent state and means that the processes are NOT instrumented. It is usually caused by the agent software having been uninstalled and reinstalled on the agent system. To force the processes to be instrumented, uncheck the Instrument? option and check it again.

Harvest Option

The Harvest option is dimmed unless the Instrument option is selected. Selecting the Harvest option has two effects:

 It causes the analysis server to retrieve traces for the processes from the agent.

 It may use a license unit, as follows: – Java processes will always use a license unit when Harvest is selected. – Only the first .NET process on an agent for which Harvest is selected will use a license unit. Selecting Harvest for additional .NET processes will not use additional units. The Harvest option is dimmed in cases where selecting it would use a license unit, but there are no units available. Unless Harvest option is selected, transaction trace data for the processes will not appear in the analysis server interface. For that reason, you typically enable both the Instrument and Harvest options. Disable Harvest if you need to release a license unit for another process to use. See “Managing License Unit Usage“ in the “License Configuration Screen“ topic for details. Clearing the Harvest option takes effect immediately. The processes do not have to be restarted.

Network Data (SaaS Analysis Server Only)

Note: Available Only with SaaS Analysis Server This feature is available only with the Software as a Service (SaaS) offering of the analysis server (SteelCentral SaaS). It is not available when the analysis server is installed in your environment (“on premises”).

Note: Not Seeing These Settings? On Windows agents, the Network Data settings appear only if the Windows system has the Npcap library installed. The library is not included as part of the agent installation. You can download the Npcap library at https://nmap.org/npcap/. After installing Npcap, restart the AppInternals agent.

These settings control whether the “NPM Sub-Agent“ collects network data. Choose “ON | OFF“ to disable or enable reporting of network data altogether.

32 Aternity APM Version 11.4.2 Agent Details Screen

Additional settings control the destination for the network data and optionally filter which traffic will be monitored:

 “Choose Network Data Destination“ options control whether to send data to APM SaaS or another SteelCentral component.

 Choose “Collect from all interfaces“ to report data from all the network interfaces that the NPM sub-agent detected.

 Choose “Collect from specific interfaces“ to limit which interfaces to report data from.

ON | OFF

The ON | OFF toggle controls whether the analysis server reports network data at all. Click the current setting to change it. When the setting is OFF, the Network Data area collapses to hide other settings. The ON | OFF toggle takes effect immediately. You do not have to click Save.

Collect from all interfaces

Choose this setting to report data from all the network interfaces that the NPM sub-agent detected. You can refine which traffic from all interfaces the analysis server reports as described in “Berkeley Packet Filter (BPF)“.

Collect from specific interfaces

Choose this option to configure reporting for specific network interfaces. (This option is useful when the NPM sub-agent detects multiple interfaces and lists them in the table.) To configure a specific interface:

1) Choose Collect from specific interfaces

2) Click the edit icon ( ) for the interface you want to configure.

3) The Edit Network Interface dialog opens. –Clear the Collect data from this interface check box to disable reporting for this interface only. – You can also specify “Berkeley Packet Filter (BPF)“ syntax that applies to this interface only.

Save / Revert

Click Save to save any changes made in the Network Data area. Click Revert to discard any changes.

Berkeley Packet Filter (BPF)

Use this text box to supply Berkeley Packet Filter syntax that filters the network traffic reported. Filters specified here apply only if the “Collect from all interfaces“ option is selected. When it is selected, any filters specified here apply to all interfaces detected by the NPM sub-agent. If “Collect from specific interfaces“ is selected, you can provide BPF filters that apply to individual interfaces. Click the edit icon ( ) to specify the filters.

Aternity APM Version 11.4.2 33 Agent Details Screen

With BPF, you can filter network traffic based on criteria such as IP addresses, protocols and port numbers. BPF is a standard method of packet filtering used by tools such as Wireshark and tcpdump. The pcap-filter man page describes the packet filter syntax used by BPF: https://www.wireshark.org/docs/man-pages/pcap-filter.html Examples:

 Match only packets coming from or going to host nhv1-rh6-2: host nhv1-rh6-2

 Match only packets coming from (src) host nhv1-rh6-2: src host nhv1-rh6-2

 Match only packets going to (dst) host aet-cldsrv-3: dst host aet-cldsrv-3

 Combine filters: Match only packets coming from host nhv1-rh6-2 and going to host aet-cldsrv-3: src host nhv1-rh6-2 and dst host aet-cldsrv-3

 Not: Match only packets that do not match the filter: not (src host nhv1-rh6-2 and dst host aet-cldsrv-3)

 Or: Match only packets coming from host nhv1-rh6-2 or coming from host aet-cldsrv-3: src host nhv1-rh6-2 or src host aet-cldsrv-3

Choose Network Data Destination

These settings control the destination for network data generated by the “NPM Sub-Agent“.

 Send summarized network data to SteelCentral SaaS is the default. This option generates summary network data that the SaaS analysis server downloads and displays on the “Network Tab“.

 The second Send: option controls whether the sub-agent sends network data to SteelCentral NPM components. (This option is not available for agent releases earlier than Version 10.16.0. ) Choose this option to configure the sub-agent as a data source for those components: – Send Packets to SteelCentral AppResponse Cloud enables the sub-agent as a data source for AppResponse Cloud. Supply the fully-qualified domain name (FQDN) or IP address of an AppResponse Cloud instance. – Send SteelFlow to SteelCentral Flow Gateway enables the sub-agent as a data source for SteelCentral NetProfiler (via Flow Gateway). Supply the fully-qualified domain name (FQDN) or IP address of a Flow Gateway. This option is supported only for agents installed on Linux. Note—If you choose either of these preceding two options, the sub-agent no longer generates data for the SaaS analysis server, and the Network tab will not display data for this agent.

34 Aternity APM Version 11.4.2 Agent Details Screen

Process Data (SaaS Analysis Server Only)

This setting controls whether the analysis server reports data from the “Processes Sub-Agent“. The ON | OFF toggle takes effect immediately. You do not have to click Save.

JMX Data

This setting controls whether the analysis server reports data from the “JMX Sub-Agent“ for Java processes that are not already monitored by the “JIDA Sub-Agent“. Processes that start with the JIDA sub-agent report JMX data even if this setting is OFF. The ON | OFF toggle takes effect immediately. You do not have to click Save. This setting controls reporting at the agent level, for all Java processes on the agent. You may also have to separately configure the Java command line in order for the JMX sub-agent to connect to the JMX server for the process. See “Configuring Java Processes to Allow the JMX Sub-Agent to Connect“ for details.

Aternity APM Version 11.4.2 35 Agent Details Screen

36 Aternity APM Version 11.4.2 CHAPTER 5 Process List Screen

Use this screen to modify multiple processes to instrument in a single operation. By default, the “List of Processes to Instrument“ shows all processes on all agents that the analysis server is monitoring. Use the “Filters“ text box to specify a string that limits the processes displayed. Use the “Actions“ buttons to select specific processes and perform an action that affects all of those processes in a single operation. You can:

 Reassign the configuration that controls details of monitoring the processes to instrument. (See “Define a Configuration Screen“ for details.)

 Start or stop instrumentation.

 Start or stop harvesting of processes that are being instrumented.

 Enable multiple JMX Custom Metrics configurations on agent processes. For more information, see “Enabling Custom Metrics Configurations for JMX“.

Filters

Use the Filters text area to specify patterns to match one or more names:

 Agent names

 Process names

 Configuration names (create configurations in the “Configurations Screen“)

 Tag key names or values (create “Tags“ in the “Agent Details Screen“) As described in “Specifying Multiple Patterns“, you can combine different pattern types in a filter to create flexible filters that match the names that you want. The filter takes effect when you press ENTER, TAB, or click in another area of the screen. Click Reset to remove all of the filter settings and show all processes in the “List of Processes to Instrument“.

Aternity APM Version 11.4.2 37 Process List Screen

Simple Patterns

Simple filter patterns match any process with a name that contains the pattern. Patterns are not case sensitive. For example, the pattern web will match all these process-configuration combinations:

Process to Instrument Configuration

AwSMPwebservice default config WebWorks.Automap default config iisexpress MyWebConfig Microsoft.VisualStudio.Web.Host default config WaIISHost MyWebConfig WebPlatformInstaller default config

Wildcards in Simple Patterns

Simple patterns can include an asterisk (*) to match zero or more characters and a question mark (?) to match a single character. Adding wildcards change the behavior from matching names that contain the pattern to a full-string match. For example, changing the pattern from web to web* will only match agent, process, or configuration names that begin with “web”. Using the same names as in the previous example, web* will match only 2 process-configuration combinations:

Process to Instrument Configuration

WebWorks.Automap default config WebPlatformInstaller default config

Add wildcards to the beginning and end of the pattern string to see similar behavior to patterns that do not have wildcards.

Property=Value Patterns

These patterns specify a property keyword and a string value as a property=value pair. The only valid operator is =. Spaces around the = operator are not allowed. Filters support the following properties:

Property Matches

agent Matches any process with on an agent whose name contains the value.

proc Matches any process name that contains the value.

config Matches any process with a configuration name that contains the value.

38 Aternity APM Version 11.4.2 Process List Screen

Property Matches

new A value of true matches processes that have not been instrumented. True is not case sensitive. False, and any other value but true, matches process that have been instrumented. tagkeyname A complete tag key name. This filter matches any process name on an agent with that server-tag key and a server-tag value that contains the value. For example, the following pattern will match processes on agents where the logical tag value contains tomcat: logical server=tomcat

You must supply the complete key name. In the previous example, specifying logical or server would not work.

Specifying Multiple Patterns

Filters can specify multiple patterns separated by spaces. Names must match all patterns specified in multiple patterns. (In other words, the filter evaluates multiple patterns in a logical AND operation.) Specifying multiple patterns provides additional flexibility to refine filters so they match only those processes on which you want to perform an action.

These Patterns Are Evaluated As

nhlab tomcat (agent=nhlab OR proc=nhlab OR config=nhlab) AND (agent=tomcat OR proc=tomcat OR config=tomcat) agent=nhlab tomcat (agent=nhlab) AND (agent=tomcat OR proc=tomcat OR config=tomcat)

Aternity APM Version 11.4.2 39 Process List Screen

Actions

This area contains buttons that select rows in the “List of Processes to Instrument“ and perform actions on selected rows.

Button Effect

Select Selects or clears selection on rows displayed in the “List of Processes to Instrument“. Click the button to display criteria by which to select rows: •All •None • Instrumentation Enabled • Instrumentation Disabled • Harvesting Enabled • Harvesting Disabled Click a criterion to select those rows. Actions Take one of the following actions on the selected processes to instrument: • Start Instrumenting: This is the same as selecting the Instrument option in the “Edit Processes to Instrument Dialog“ for the selected processes. See the “Instrument Option (Restart Required)“ for more details. • Stop Instrumenting: This is the same as clearing the Instrument option in the “Edit Processes to Instrument Dialog“ for the selected processes. See the “Instrument Option (Restart Required)“ for more details. • Start Harvesting: This is the same as selecting the Harvest option in the “Edit Processes to Instrument Dialog“ for the selected processes. See the “Harvest Option“ for more details. Similar to the Edit Process to Instrument dialog, this action will only apply to processes that are already being instrumented. • Stop Harvesting: This is the same as clearing the Harvest option in the “Edit Processes to Instrument Dialog“ for the selected processes. See the “Harvest Option“ for more details. This button is disabled until you select a row in the table. Change Configuration Assigns the configuration you choose from the list to selected rows. This button is disabled until you select a row in the table. Click the button to display the names of saved configurations (in other words, the same list as the “Configurations Screen“ shows). Click a configuration name to assign the selected processes to instrument to that configuration.

40 Aternity APM Version 11.4.2 Process List Screen

List of Processes to Instrument

The list of processes is presented as a table. Click on a column heading to sort the column in ascending order based on the column value. Click the column heading again to sort in descending order. An icon indicates ascending or descending sort order.

Column Meaning

Selected (check box) Select the check box to select the row so it is affected by the “Actions“ you choose.

Agent The “Agent Status“ and name of the agent on which processes are running. Click the name to open the “Agent Details Screen“ for that agent. Tags Any tags defined for the server. As described in “Tags“, tags are custom identifiers created by administrators to categorize servers that AppInternals is monitoring. Create tags in the “Agent Details Screen“. If there are any tags (including any automatically generated “Docker Container Tags“), this column shows the first tag value (but not the tag name). If there are multiple tags, click the expander to see all values. Pause the mouse pointer over the value to see the tag name and value.

Processes to Instrument The processes-to-instrument name. This is the same as the “Processes to Instrument“ column in the “Agent Details Screen“. Configuration The name of the “configuration” with settings that control what application data that JIDA or the dotNet sub-agent collects for the processes-to-instrument name. Click the name to open the “Define a Configuration Screen“ for the applicable configuration. Instrument Whether the processes will load (or unload) the APM monitoring library when it next restarts. This is the same value as the “Instrument“ column in the “Agent Details Screen“. Harvest This is the same value as the “Harvest“ column in the “Agent Details Screen“.

Status The instrumentation status for the corresponding processes. This is the same value as the “Status“ column in the “Agent Details Screen“.

“show / hide processes of offline agents that released licenses”

This option appears when both the following conditions are true:

 The analysis server is monitoring agents that are configured to release license units when they are offline.

Aternity APM Version 11.4.2 41 Process List Screen

 Any of those agents are “Offline“. By default, processes discovered on such agents do not appear in the “List of Processes to Instrument“. In cloud environments, agent virtual machines can be short-lived and release licenses when they terminate. There may be many such agents and it is typically not useful to include their processes in the list. Click show processes of offline agents that released licenses to display such agents. The option changes to hide processes of offline agents that released licenses. Click it to again hide them. 9/14/2017: Amrinder reply to email: Starting in 10.10, the agents should be removed from the list if they have been offline for more than 30 days. That value can be tweaked using the cli agent_setting command. After an agent has been offline for more than 30 days, the analysis server removes its entry altogether and it will not appear even when you click show offline agents that released licenses. See these related topics:

 ““show / hide offline agents that released licenses”“ option in the “Agent List Screen“

 “Retain / Release Licenses on Disconnect“ option in the “Agent Details Screen“

 “Deploying Agents in Dynamic Environments“

 “Managing License Unit Usage“

42 Aternity APM Version 11.4.2 CHAPTER 6 Configurations Screen

This screen lists saved configurations. Each configuration controls what application data that JIDA and the dotNet sub-agents collect on agents. The settings for a particular configuration apply only to processes to instrument that are assigned to the configuration. Until you create additional configurations, all processes to instrument are assigned to the configuration named default config.

 Click Define a Configuration to create a new configuration and open the “Define a Configuration Screen“. New configurations are automatically named config - x. You can rename them in “Current Configuration“ area of the Define a Configuration screen.

 You can limit the configurations listed by choosing one or more of the criteria to the left. Each criterion limits the configurations to those that have matching processes to instrument.

 You can also limit the configurations listed by typing a a filter pattern in the text box. The pattern matches configuration names. Patterns can use an asterisk (*) as a wildcard to match zero or more characters and a question mark (?) to match a single character. Patterns are case sensitive. The pattern matching takes effect as you type.

 Sort the list by choosing a Sort by option.

 Click a configuration name to open an existing configuration in the “Define a Configuration Screen“.

Aternity APM Version 11.4.2 43 Configurations Screen

44 Aternity APM Version 11.4.2 CHAPTER 7 Define a Configuration Screen

Overview

This screen has settings that control monitoring for a “configuration”, including:

 Whether APM collects web page performance (“end-user experience”) data

 Which application classes JIDA and the dotNet sub-agents monitor

 Advanced settings such as disk usage limits and logging The settings for a particular configuration apply only to processes to instrument that are assigned to the configuration. By default, all processes to instrument are assigned to the configuration named default config. You change the assignment for processes to instrument in the “Process List Screen“ or in the “Agent Details Screen“. Settings on this screen control how APM monitors processes to instrument. They do not control whether they are monitored. To enable monitoring, use the “Edit Processes to Instrument Dialog“ in the “Agent Details Screen“ or the Start Instrumenting action in the “Process List Screen“.

Click Save After Making Changes!

You must click the Save button at the bottom of the screen for any changes to take effect. Navigating from the screen by clicking another menu option will discard any changes you have made. For changes that affect whether data appears in the APM interface, allow 5 minutes for data to propagate from the agent to the analysis server.

How Settings Here Control Data That APM Collects

APM collects data from a variety of sources. This screen has settings that affect some of those sources:

Aternity APM Version 11.4.2 45 Define a Configuration Screen

 Web page: APM monitors web page performance by collecting data on page loads and (if configured) AJAX requests in users’ web browsers. This data reflects application performance from the perspective of the end user. The “Collect End-User Experience Data“ option controls whether APM collects this data. The “Customize Snippet“ settings control details of that collection.

 Application: APM monitors application performance at a JVM (for Java) or appdomain (for .NET) level. JIDA and the dotNet sub-agent instrument classes and methods they are configured to monitor. By default, they collect data as follows: – For “interesting” methods in application classes. This default is specified by selecting the Normal preset monitoring level (see “Data Collection Settings: Preset Levels“). You can control which application methods are collected by choosing the Custom preset level and creating a custom monitoring filter (see “Monitoring Filters“). – Categories of Java and .NET classes that are typically of interest and that the sub-agents can identify automatically (for example, database connectivity (Java JDBC or .NET ADO) or web services). These categories monitor specific classes and methods that are not configurable. However, for troubleshooting purposes, you can disable collection entirely for some categories in the “Optional Data to Report“ settings.

 Environment: APM monitors key operating system resources metrics such as CPU, memory, and networking on systems where agents are installed. This monitoring is automatic and requires no configuration. None of the settings on this screen affect environmental data.

Settings That Require Restarting Processes

Most settings on this screen are dynamic and do not require restarting the processes assigned to this configuration. However, changing settings that affect which classes that APM sub-agents collects data for will sometimes require restarting processes. Specifically, changing these settings may require restarting:

 Adding or modifying custom “Monitoring Filters“.

 Enabling or disabling collection of categories in “Optional Data to Report“ and “Optional Data to Report (Java)“. When you change any of these settings, the “Status“ column in the “Agent Details Screen“ for affected processes changes to May Require Restart. Keep in mind the following:

 Restarting is NOT required when you DISABLE collection: – Add or modify a custom filter and set “Collect (Interesting Only, Always, Never)“ to Never. – Clear (uncheck) any of the options in “Optional Data to Report“ and “Optional Data to Report (Java)“ The fact that you do not have to restart processes when disabling settings is useful for “Troubleshooting Application Problems Caused by APM Monitoring“.

 Restarting typically IS required when you ENABLE collection: – Add or modify a custom filter to collect previously (since the last restart) uncollected classes and methods. – Select (check) any of the options in “Optional Data to Report“ and “Optional Data to Report (Java)“.

46 Aternity APM Version 11.4.2 Define a Configuration Screen

The only time restarting is not required is if all the methods that match the new or changed filters were already collected. You can determine which methods are collected by scanning log files as described in “Reviewing Log Files to See Classes and Methods that Are, Are Not Instrumented“.

Reporting Method Parameter Values in Transaction Traces

Configurations can specify that transaction trace data include parameter values for methods that are being monitored:

 Custom “Monitoring Filters“ can specify specific parameters to report in the “Parameters“ setting.

 Under “Data Collection Settings: Custom“, the “Parameter Reporting“ settings enable or disable parameter reporting for specific categories of classes.

Current Configuration

This area shows details about the configuration:

 “Configuration Name and Renaming“

 “x processes to instrument using this configuration.“

 The user that created the configuration

 When the configuration was last modified

Configuration Name and Renaming

The name identifies settings for a particular configuration in the “Process List Screen“ and “Agent Details Screen“. Rename the configuration by supplying a new name in the text box. The change does not take effect until you click Save. (You cannot rename the default config configuration.)

Edit

Click Edit to show options that affect the configuration as a whole.

Download

Click Download to generate a file with the configuration’s settings and download it to your local system. This is primarily useful for configuring an agent that will be deployed in a cloud environment. In such environments, you configure the agent so that processes you want to monitor will be automatically instrumented when they first start on the agent system. This avoids manual configuration in the APM interface and having to restart processes. As described in “Configuring Processes Running in PaaS Environments to Be Instrumented on Initial Startup“ in the “Deploying Agents in Dynamic Environments“ topic, you specify the processes to automatically instrument in a mapping file. The mapping file also specifies a corresponding configuration files with settings for the processes. Use the Download option to create the configuration files.

Aternity APM Version 11.4.2 47 Define a Configuration Screen

The file names are based on the configuration name, using the form appinternals-_.json:

 Spaces and periods are replaced with underscores.

 Any characters in that are illegal in file names (such as ?\/:) are removed.

Clone (Copy)

Click Clone to create a copy of the configuration. The copy is named originalname - copy.

Delete

Click Delete to delete the configuration. (You cannot delete the default config configuration.) If there are no processes to instrument assigned to this configuration, then the configuration is simply deleted. However, if any processes to instrument are assigned to this configuration (as indicated by the message below the delete button), then those processes are assigned to the default config configuration. A dialog opens so you can confirm that you want to go ahead with the delete and reassignment operation. Note that if any of the processes implicitly reassigned to the default config configuration are already instrumented, they may have to be restarted before any differences in the default config configuration will take effect. x processes to instrument using this configuration.

This indicates how many processes-to-instrument names are assigned to this configuration. To see specifically which names, click the link to open the “Process List Screen“ filtered to show those names.

“High-Overhead Settings” Banner

This banner appears when you specify custom settings that potentially affect application or system performance:

 “Monitoring Filters“ that match many methods (see “Avoid Creating Broad Filters“)

 “Enable Verbose logging“

 “Include application exceptions“

“Overrides Enabled” Banner

This banner appears in the Current Configuration area if the configuration includes any “Configuration Overrides“. Be aware that any configuration overrides take precedence over conflicting settings specified elsewhere in this screen.

48 Aternity APM Version 11.4.2 Define a Configuration Screen

Data Collection Settings: Preset Levels

The preset collection levels provide increasing levels of performance monitoring. Typically, the default Normal level provides rich application data with minimal overhead. Pick another level to increase or reduce the data collected, or pick Custom to specify all settings manually. To see the specific properties affected by each level, choose a level and click Save. Reopen the configuration and click Custom. (The table below also shows this detail.) Choose from the following options:

 Collection Off: Disables all collection settings. This is useful for “Troubleshooting Application Problems Caused by APM Monitoring“.

 Low: Enables monitoring of common application entry and exit points.

 Normal: Enables all Low settings, plus monitoring of additional application entry points, collection of end-user experience data, and monitoring of all “interesting” application methods. Normal is the default collection level.

 High: Enables all Normal settings, plus parameter reporting for some categories of classes, and reporting of all HTTP cookies and headers.

 Custom: Displays individual collection settings to customize monitoring. Use Custom if you need to vary from the preset levels. See “Data Collection Settings: Custom“ for details on those settings.

Note: Interaction of Preset and Custom Settings When you click Save, the analysis server saves the settings as currently displayed. For example, if you choose Custom and change settings, those changes are retained if you click a preset level, provided you click Custom again before clicking Save. If, however, you click a preset level and click Save, the analysis server discards any custom changes you may have made.

The following table shows the specific custom settings that correspond to each preset level. ✕ means the setting is disabled and ✓ means it is enabled:

Setting Collection Low Normal High Off

“Thread“ ✕ ✓ ✓ ✓

“Garbage Collection“ ✕ ✓ ✓ ✓ “Web“ ✕ ✓ ✓ ✓ “Web Services“ ✕ ✓ ✓ ✓ “Message Bus (OSB and ASB)“ ✕ ✓ ✓ ✓ “Database (JDBC and ADO)“ ✕ ✓ ✓ ✓ “Remote“ ✕ ✓ ✓ ✓

“Enterprise JavaBeans (EJB)“ ✕ ✓ ✓ ✓ “Java Message Service (JMS)“ ✕ ✓ ✓ ✓ “Collect End-User Experience Data“ ✕ ✕ ✓ ✓

Aternity APM Version 11.4.2 49 Define a Configuration Screen

Setting Collection Low Normal High Off

“Java Naming and Directory Interface (JNDI)“ ✕ ✕ ✓ ✓ “Remote Method Invocation (RMI)“ ✕ ✕ ✓ ✓ “Portlets“ ✕ ✕ ✓ ✓ “Monitoring Filters“ Never Never Interesting Interesting “SQL Bind Variables“ ✕ ✕ ✕ ✓ “HTTP Cookies/Headers“ ✕ ✕ ✕ Report all “Enable End-User Experience Tracing Cookie ✕ ✕ ✕ ✓ (legacy)“ “Report Web Tier Parameters“ ✕ ✕ ✕ ✓ “Report RMI Parameters“ ✕ ✕ ✕ ✓ “Report EJB Parameters“ ✕ ✕ ✕ ✓ “Maximum string length“ n/a 200 200 5000 “Maximum write rate“ n/a 10 KB 20 KB 50 KB

Data Collection Settings: Custom

These settings appear when you click the Custom option in the “Data Collection Settings: Preset Levels“. Typically, you do not need to change these settings.

Collect End-User Experience Data

Select this option to monitor web application performance from the perspective of end users. When you select this option, JIDA and the dotNet sub-agents add a JavaScript snippet to web pages that they monitor. The script provides data to APM about page response time, traffic, geography, platform type, and other measurements. For every transaction that agents monitor, users’ web browsers send corresponding web page performance data to the analysis server. This setting is dynamic and does not require restarting the processes assigned to this configuration. This option also causes JIDA and the dotNet sub-agent to automatically add the JavaScript snippet to web pages that the jobs assigned to this configuration are monitoring.

 JIDA injects the JavaScript into servlet and JSP pages.

 The dotNet sub-agent injects the JavaScript into ASP.NET pages. NOTE: This feature does not work in .NET Version 1.1.

Enabling This Option Affects Analysis Server Performance

Processing data generated from web pages increases the overhead on the analysis server. Enabling this option reduces the number of stitched transactions the analysis server can process by roughly 30 percent.

50 Aternity APM Version 11.4.2 Define a Configuration Screen

See the System Requirements (available from the APM support page) for more details about the performance impact of enabling this option.

View Snippet

Click this button to open a separate dialog that displays the JavaScript snippet. This is useful for the following purposes:

 Copy the snippet to add to web pages that APM agents are not monitoring.

 Verify that changes you specified in the “Customize Snippet“ options appear as expected.

Customize Snippet

Click this link to display the “Customize Snippet“ options and change behavior of the snippet. Typically, you do not need to customize the snippet.

Monitoring Filters

These settings appear only when you choose the Custom preset level in the “Data Collection Settings: Preset Levels“ control. They control custom filters that specify arbitrary class and methods in your application that you want to monitor. APM reports these classes under the “Application Code“ category described in the “Categories Reference“ topic. (By comparison, the “Optional Data to Report“ and “Optional Data to Report (Java)“ settings enable or disable special predefined categories of classes.) If you choose the Normal preset level (see “Data Collection Settings: Preset Levels“) these settings are not visible. Instead, APM collects data for most application classes and methods (public only) that it determines are “interesting”. When instrumentation is enabled, APM automatically detects these classes and methods and monitors them without any further configuration. The default automatic discovery and monitoring of interesting application classes and methods typically works without problem and requires no manual configuration. You can override this default behavior by creating custom monitoring filters as described in this section. Use the “Edit Filter Dialog“ to add or modify monitoring filters. Monitoring filters selectively override automatic discovery and monitoring. They specify classes and methods of interest and the corresponding level of monitoring that you want. Cases where you may need to change the default behavior include:

 You want to monitor methods that are not otherwise reported. As described in “Monitoring Levels for Custom Filters“, different settings monitor additional methods.

 You want to suppress monitoring specific classes and methods that are otherwise automatically reported. As described in “Monitoring Levels for Custom Filters“, the Collect Never setting suppresses monitoring of matching classes and methods. This is useful if instrumentation causes a problem with your application (see “Troubleshooting Application Problems Caused by APM Monitoring“ for details).

 You want to report method parameters and values for certain methods. See “Reporting Method Parameter Values“ for details.

Aternity APM Version 11.4.2 51 Define a Configuration Screen

Monitoring Levels for Custom Filters

You choose the monitoring level for a custom filter in the “Collect (Interesting Only, Always, Never)“ setting in the “Edit Filter Dialog“. Choose one of the following levels:

 Interesting Only: monitor only those methods that are interesting, as described in “What Are Interesting Methods?“.

 Always: monitor the following additional methods: – Methods that are not interesting. Always bypasses the “interesting” methods check described in “What Are Interesting Methods?“, so monitors any additional methods that check would have excluded. –Private methods. In addition, Always filters where the “Method“ setting specifies a name with no asterisk wildcard characters have special meaning. They are interpreted as “force monitor” filters where the matching method will be monitored if possible. See “Force Monitor: “Always” Filters that Specify an Explicit Method“ for details. However, note that Always does not mean that all methods will be monitored. See “Methods That Are Never Monitored“.

 Never: do not monitor any methods. The following table summarizes the monitoring levels for custom filters and which methods are monitored at each level. ✕ means the methods are not monitored and ✓ means they are.

Method Never Interesting Always “Force Only Monitor”

Meets the “interesting” criteria ✕ ✓ ✓ ✓ Does NOT meet the “interesting” criteria ✕ ✕ ✓ ✓ Is private ✕ ✕ ✓ ✓ Is shorter than 6 bytes ✕ ✕ ✕ ✓ Was detected as a “hotspot” method ✕ ✕ ✕ ✓ Is abstract, native or a constructor ✕ ✕ ✕ ✕ Makes a tail call ✕ ✕ ✕ ✕

Never Never ✕ ✕ ✕ ✕

monitored Part of mscorlib.dll

What Are Interesting Methods?

APM considers methods “interesting” if they meet ALL of the following criteria:

 For .NET, call another assembly

 For Java, call another package (other than the java .* package)

 Contain loops

 Are long (a few hundred instructions)

52 Aternity APM Version 11.4.2 Define a Configuration Screen

 Are NOT: – Internal methods (Java or .NET language, internal application server, or APM' own) – For .NET, methods in any assembly with System.Reflection.AssemblyProductAttribute set to Microsoft\xc2\xae .NET Framework – For Java, methods whose containing class is loaded by ExtClassLoader and BootClassLoader – For Java, methods whose containing class is loaded by AppClassLoader , IF the Java process is detected to be one of the following application servers: WebSphere, Tomcat, WildFly, WebLogic Separately, for .NET, methods are considered interesting if they are .NET events and callback methods (such as btnPurchase_OnClick and Page_OnLoad), even if they do not meet the other “interesting” criteria.

Force Monitor: “Always” Filters that Specify an Explicit Method

Filters with the following settings are interpreted as “force monitor” filters where the matching method will always be monitored if possible:

 The “Collect (Interesting Only, Always, Never)“ setting is Always.

 Both the “Class“ and “Method“ settings do not specify any asterisk wildcard characters. Specifically, such filters will monitor the matching method even if it is short (less than 6 bytes) and even if it was detected as a hotspot. See the “Hotspot Detection“ section of “Instrumentation Techniques and Troubleshooting“ for details on hotspots.

Note: Exercise care in using force-monitor filters Hotspot detection is an important technique to reduce the overhead of instrumentation. Forcing instrumentation of hotspot methods increases instrumentation overhead.

You can confirm that JIDA or the dotNet sub-agent instrumented the method as “force monitor” from messages in the log files. For example, consider a filter with the following settings in the “Edit Filter Dialog“: Class: org.riverbed.test.special.SpecialHotSpot Method: hotBabyHot Look in log file for the Added collect_generic filter message with the method name (hotBabyHot in this example) and forcetrace=true: 01/05/2018 03:35:48 PM, JIDA , -1252337920 , INFO , Added collect_generic filter: class=org.riverbed.test.special.SpecialHotSpot, method=hotBabyHot, parameter=, metric=false, metriclevel=method, forcemetric=false trace=true, forcetrace=true, interestingonly=false

A warning message in the log shows that the method was actually instrumented even though it was detected as a hotspot: 01/05/2018 03:36:00 PM, JIDA , -1269745920 , WARN , Hotspot detected in method org.riverbed.test.special.SpecialHotSpot.hotBabyHot()V 147389 calls took only 99000 microseconds. Call rate=240 - This method was not hot-spotted because it matched a collect filter (forcetrace). Consider removing the filter.

Agent and sub-agent log files reside in the /Panorama/hedzup/mn/log directory on the agent system.

 Log file names for the dotNet sub-agent start with the prefix DA-dotNet.

Aternity APM Version 11.4.2 53 Define a Configuration Screen

 Log file names for the JIDA sub-agent start with the prefix DA-JIDA.

Methods That Are Never Monitored

Keep in mind that APM does not instrument the following methods even if they meet the interesting criteria or match a monitoring filter that specifies Always in the “Collect (Interesting Only, Always, Never)“ setting:

 Methods that are abstract, native, or constructors.

 Methods that make a tail call (see http://en.wikipedia.org/wiki/Tail_call).

 For .NET, base framework methods which reside in the core runtime library, mscorlib.dll.

Edit Filter Dialog

Use the Edit Filter dialog to create or modify custom monitoring filters that control which classes and methods are monitored and details of parameters to report.

1) Click Add a Filter to create a new filter. Click the edit icon ( ) in the row for an existing filter to modify that filter.

2) Specify the following settings: – “Collect (Interesting Only, Always, Never)“ specifies the level of monitoring for the filter. – “Class“ specifies a wildcarded-string that limits which classes the filter applies to. – “Method“ is optional and limits which methods within the matching classes the filter applies to. – “Parameters“ is optional and specifies method parameters to report with transaction details.

3) Click Save to save the filter and close the Edit Filter dialog. Note that the Save option is disabled if the strings specified in the Class and Metric settings match any existing filters.

4) Click Save at the bottom of the screen to save the configuration.

Collect (Interesting Only, Always, Never) This setting specifies the level of monitoring for matching classes and methods. See “Monitoring Levels for Custom Filters“ for details.

Class This setting specifies a wildcarded-string that limits which classes the filter applies to. Review _methods.txt files to find candidate classes and methods for this setting (see “Reviewing Log Files to See Classes and Methods that Are, Are Not Instrumented“). Values are case sensitive. You can use an asterisk (*) as a wildcard character at the beginning and end of the value to potentially specify multiple classes. Wildcards are not allowed within the value. Valid: com.TradeFast.SecureService.StockRequestHandler com.myco.foo.Bar* PaintDotNet.SystemLayer PaintDotNet.Rendering.* PaintDotNet.*

54 Aternity APM Version 11.4.2 Define a Configuration Screen

Not valid (wildcard in middle of the class specification): com.myco.*.Bar

Method This setting specifies a comma-separated list of methods to monitor for the classes that match the “Class“. If you do not supply a value here, the default is to monitor all methods in the class (except for “Methods That Are Never Monitored“). Values are case sensitive. The setting allows an asterisk at the beginning and end of any method name within the list.

Note: Specify the complete method name without wildcards to force monitoring of the method. See “Force Monitor: “Always” Filters that Specify an Explicit Method“ for details.

Valid: *Amortization,GetSecurities*,GetWeather*

Not valid (wildcard in middle of method name): GetSecur*Weather

An alternative to specifying a list of methods is to specify multiple filters with the same “Class“, but that each specify a different method. The following example uses the class specification com.tradefast.*.

Class Method

com.tradefast.* *Amortization com.tradefast.* GetSecurities* com.tradefast.* GetWeather*

Parameters This setting specifies method parameters whose values you want to be reported. Transaction traces will include the parameter values and names for methods that match the classes and methods the filter specifies. “Reporting Method Parameter Values“ describes this setting in detail.

Reviewing Log Files to See Classes and Methods that Are, Are Not Instrumented

Before you add custom filters, review which methods are already instrumented and being monitored. The JIDA and dotNet sub-agents write a list of instrumented methods to the files DA-dotNet__methods.txt and DA-JIDA__methods.txt in the /Panorama/hedzup/mn/log directory on the agent system. IN messages in these files show every method that the sub-agent instrumented and why. In the following example, the T: portion of the message indicates the type of instrumentation. Here, the method was automatically instrumented because it was a servlet, which matches the “Web“ category: IN 8698 c:filters.ExampleFilter m:doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/Servl etResponse;Ljavax/servlet/FilterChain;)V |V T:Special:servletfilter (ServletFilter)

If these methods cause problems, they may be candidates for monitoring filters that specify Never in the “Collect (Interesting Only, Always, Never)“ option.

Aternity APM Version 11.4.2 55 Define a Configuration Screen

To see which methods the sub-agent discovered but that it did not instrument, enable the “Include all discovered methods (Restart Required)“ option. This option writes additional messages in the files for discovered methods that were not instrumented. These message begin with a hyphen ( - ). In the following example, the R: portion of the message indicates the reason the method was not instrumented. Here, the method was not instrumented because it did not meet the “interesting” criteria (see “What Are Interesting Methods?“): - 8698 c:filters.ExampleFilter m:toString()Ljava/lang/String; R:no loop or call to another class

These methods may be candidates for monitoring filters that specify Always in the “Collect (Interesting Only, Always, Never)“ option. Access log files in any of the following ways:

 Use a web browser. The agent serves log files at this URL: http://:2111/log/

 Create and download a “diagnostic bundle” that includes the log files. On the “Agent Details Screen“, click the “Download the agent logs and diagnostic files“ link.

 Edit the log files directly in the /Panorama/hedzup/mn/log directory.

Avoid Creating Broad Filters

Any filter that specifies Always and a Class filter string that matches many methods will likely cause problems, including the following:

 Long application startup times

 Monitoring of uninteresting methods

 Reduced detail in transaction traces

 Transactions incorrectly grouped as related Avoid creating filters with inclusive wildcard strings such as * or com* and a “Collect (Interesting Only, Always, Never)“ setting of Always.

Precedence of Custom Filters

As APM discovers application methods, it compares them with filters in the list and uses the first class and method combination that matches. For this reason, the filter list automatically puts more specific filters first, following these rules:

1) Method lists without wildcards.

2) Method lists with wildcards.

3) Class specifications without wildcards.

4) Class specifications with wildcards.

5) Filters that are prefixes of other filters follow those filters: Com.riverbed.client.ui.* Com.riverbed.client.*

Com.riverbed.*

6) Class specifications that begin with a a wildcard.

56 Aternity APM Version 11.4.2 Define a Configuration Screen

7) The default wildcard class specification of *.

Custom Filters Usually Require Restarting Applications

Whenever you add or modify a custom filter, APM changes the “Status“ column in the “Agent Details Screen“ for affected processes to instrument to May Require Restart. In almost all cases, you must restart the processes. See “Settings That Require Restarting Processes“ for more details.

Interaction of Superclasses and Subclasses May Cause Unexpected Results

In some cases, you may see methods reported that are inconsistent with monitoring filters. (Method-level detail is visible in the “Call Tree“ section of the “Transaction Details“ window.) For example, the call tree may show data from:

 Methods in classes that do not match any class specified in the “Class“ setting for a filter (or belong to a predefined categories of classes)

 Methods in classes that do match a monitoring filter that specifies Never in the “Collect (Interesting Only, Always, Never)“ setting Typically, this behavior is caused by how APM applies monitoring filters to superclasses and subclasses. In general, the filter must specify the class where the instrumented method is defined, not the class where that method is executed. However, APM will report the class name where the method is executed (the “runtime type”) as the class name, even if a method is monitored because it was defined in a superclass. Consider the following Paint.NET classes and methods. PaintDotNet.SystemLayer.ToolStripEx is a superclass that defines the method add_RelinquishFocus. Another class, PaintDotNet.Controls.ToolConfigString, derives from ToolStripEx:

Methods Reported for Subclasses That Do Not Match Filters If the class specification in a filter matches a superclass, APM may report data from methods in its subclasses, even if the subclass does not match a class specification. For example, suppose a monitoring filter that specifies Interesting Only in the “Collect (Interesting Only, Always, Never)“ setting matches the superclass: PaintDotNet.SystemLayer*

Aternity APM Version 11.4.2 57 Define a Configuration Screen

APM will monitor the add_RelinquishFocus method in the PaintDotNet.SystemLayer.ToolStripEx class because it is interesting and matches the specified filter.

Whenever Paint.NET executes the add_RelinquishFocus method, APM reports data for it. However, if the method is executed from the derived class, PaintDotNet.Controls.ToolConfigString, it will be reported under that class even though it does not match the monitoring filter.

Methods Reported for Subclasses Match Filter with ‘Never’ Collect Setting Similarly, if the class specification in a filter matches a subclass, APM may report data from its methods even if the filter specifies Never in the “Collect (Interesting Only, Always, Never)“ setting. For example, suppose the class specification in the Never filter matches the subclass: PaintDotNet.Controls.ToolConfigString

Despite the Never filter on the subclass, APM will still report data for the add_RelinquishFocus method. This is because the method is interesting and defined in a superclass, PaintDotNet.SystemLayer.ToolStripEx, that does not match the Never filter. However, the method will be reported under the derived class name, PaintDotNet.Controls.ToolConfigString, that does match filter.

To prevent reporting of methods in such cases, create Never filters that match superclasses as well as subclasses.

58 Aternity APM Version 11.4.2 Define a Configuration Screen

Reporting Method Parameter Values

Transaction traces generated by JIDA or the dotNet sub-agents can include parameter values for methods that are being monitored. Specify parameters of interest in the “Parameters“ setting of the “Edit Filter Dialog“. This section describes how to configure the Parameters setting. (The configuration described here is distinct from the “Parameter Reporting“ settings in “Data Collection Settings: Custom“. Those settings also control method parameter reporting, but affect only the corresponding predefined category of application classes.)

Quick Start: Parameters in a Custom Filter

This example shows creating a custom filter that reports string parameter values for a .NET method. See “Syntax for Specifying Parameters“ for details on additional options. You need to be familiar with the method signature to specify which parameters you want to report in transaction trace data. Consider the following excerpt from a C# class: namespace com.riverbed.test.feature.param.multi.args { public class MultipleArgTests { public void threeString(string eins, string zwei, string drei) { . . .

The threeString method has three parameters, eins, zwei, and drei. Follow these steps to report those parameter values:

1) Click Add a Filter to open the Edit Filter settings dialog and create a new filter to monitor the threeString method.

2) Choose Always in the “Collect (Interesting Only, Always, Never)“ setting.

3) Specify com.riverbed.test.feature.param.multi.args.MultipleArgTests in the “Class“ setting.

4) Specify threeString in the “Method“ setting.

5) In the Parameters setting, specify the parameters you want to report as a comma-separated list of integers representing the position of the parameter in the method signature. 1 identifies the first parameter, 2 identifies the second, and so on. 0 identifies the method return value. You can also specify

Aternity APM Version 11.4.2 59 Define a Configuration Screen

* as a wildcard for all parameters not covered by explicitly-specified positions. For this example, specify 1,2,3. The Edit Filter dialog box looks like this:

6) Click Save in the dialog box to save the changed filter. The Method column for the filter now shows (Parameters) after the method name to indicate that you configured parameter reporting:

7) Click Save in the Define a Configuration screen to save the configuration.

8) Restart the application.

How Parameters Appear in the User Interface

When trace data includes method parameters, you can search for them using the “parameter“ search field in the “Search Tab“:

60 Aternity APM Version 11.4.2 Define a Configuration Screen

The “Call Details“ area in the “Transaction Details“ window shows parameter values and names. (Click the icon for a row in the Search tab to open the Transaction Details window.)

Syntax for Specifying Parameters

The Parameters setting specifies a list of parameters whose values to report, using this syntax: { param[:param-label][> | <] }[,...] The Parameters setting allows an asterisk (*) in place of the entire list or a single element in the list. param Identifies a single parameter. param can be one of the following:

 An integer representing the position of the parameter in the method signature: – 1 through N represents the position of the parameter from left to right: 1 identifies the first parameter, 2 identifies the second, and so on. For example: 1,2,3 The APM interface displays the parameter values using the corresponding parameter names: name=, name=, and so on. For example: drei=drei zwei=zwei eins=eins – 0 specifies the method return value. The APM interface displays the return value as retval=. For example: retval={_items=["ArrayList: first String","ArrayList: second String","ArrayList: third String",null],_size=3,_version=3,_syncRoot=null} The order of the integers does not matter. The following example specifies the return value and the first and third parameters of any matching methods: 3,1,0

Aternity APM Version 11.4.2 61 Define a Configuration Screen

 The method parameter name. Specifying the name works only with the dotNet sub-agent. The dotNet sub-agent detects parameter names for methods it has instrumented by examining .NET assemblies. JIDA does not support specifying parameter names. In the APM interface, the parameter value is identified by the same name. For example: drei,zwei,eins

 A reference to an object field of a parameter using the notation param.. Use this notation to report the value of object fields and extract specific data from a method parameter value. These references can report multiple values for the same field name. For example: 1.foo,1.bar,1.child,2.parent,2.parent.child See “Examples: Member Fields of Complex Object Parameters“ for detailed examples using this notation. (Only references to object fields are allowed. References to methods and .NET properties are not allowed. Also, the notation does not support references to array elements.)

 A reference to a specific element of an array using the notation param[elementnum], where elementnum is 0 for the first element, 1, for the second, and so on. Surround the element number name with square brackets. To report values for the entire array (subject to “Limits on Parameter Value Sizes in Traces“), simply specify 1. For example, here is how an entire string array is reported: p1=["string0","string1","string2","string3","string4"] Use this syntax to report the value of the fourth element: 1[3] This syntax will result in: p1[3]="string3"

 A reference to a specific string key of a map (also called a dictionary) using the notation param{}, where param is of type map. Surround the map-key name with curly braces. For example, if the first parameter is a map with a string key named foo, use this syntax: 1{foo} This syntax will report the value associated with the key foo. For example, if the value for key foo is bar: p1{foo}="bar" The key name and value are searchable and reported in the Call Details area of the transaction details window:

62 Aternity APM Version 11.4.2 Define a Configuration Screen

This type of reference works only with string keys. Other types are ignored.

:param-label Overrides the label used to identify the parameter or return value in the transaction trace file. This can be useful with the integer format of param to supply a more meaningful name in the transaction trace: 1:reportName,3:reportID

If you want to report parameter values for all parameters but only supply a custom label for some of them, use an asterisk (*) as a wildcard value in place of an element in the list. This example overrides the label for the first method parameter but accepts the default for others: 1:reportName,*

The label values appear in the “Call Details“ in the “Transaction Details“ window and also affects searching using the “parameter“ search field. Overriding the default is useful for Java method parameters, which by default the JIDA sub-agent identifies as p1=, p2=, and so on. Use a label value to supply a more meaningful name. The dotNet sub-agent, unlike JIDA, detects method parameter names and uses them as the default label. You can still override the default name, but it is typically less useful for .NET parameters. You can also supply a label value for return values (position 0). This value overrides the default for both Java and .NET, where the return value is identified as retval=. 0:myReturnValue,*

Aternity APM Version 11.4.2 63 Define a Configuration Screen

> | < The > and < syntax uses the parameter label (if specified) and value for trace file fields that APM relies on to “stitch” related transactions into a set. In some environments, this enables APM to detect related cross-tier transactions (transactions running in another Java or .NET instance) when it otherwise could not. This syntax is never required and may disrupt cases where automatic detection of related transactions is otherwise working. However, this specialized approach can work if you are familiar with the application code and know that the application passes a unique identifier of some kind between tiers. In such an environment:

 Specify the > syntax for the parameter of the “outbound” method that calls another tier. When the method executes, the presence of > causes the sub-agents to write the SENTREQUESTINJECTEDHEADER field to the trace file and uses the parameter label and value as its value. For example, the following filter configuration specifies > after the parameter label correlationID:

When the SendMessage method executes, this configuration might generate the following trace file field (visible in the “Call Details“ in the “Transaction Details“ window): sentrequestinjectedheader: correlationID=9BC37963C7F4EE68741385057017265

 Specify the < syntax for the corresponding “inbound” method parameter on the other tier. When the method executes, the sub-agents write the RECEIVEDREQUESTINJECTEDHEADER field to the trace file and uses the parameter label and value as its value. For example, this configuration:

Might generate the following when the ReceiveMessage method executes: receivedrequestinjectedheader: correlationID=9BC37963C7F4EE68741385057017265 APM uses these trace fields to identify related transactions. The presence of a common value in the different trace files allows it to combine the traces as a related set. See “Example: Custom Stitching Using the > < Syntax“ for a detailed example.

Examples: Parameters, Labels, and Return Values

Based on the following pseudocode, “dotNet Data Adapter Examples“ and “JIDA Examples“ show the effect of different Class, Method, and Parameters values in the “Edit Filter Dialog“. namespace/package Acme.Report { public class ReportGenerator {

64 Aternity APM Version 11.4.2 Define a Configuration Screen

public boolean CreateReport(String reportName, String options, int reportID) { } public boolean DeleteReport(int reportID) { } public boolean PrintReport(String reportName) { } } } dotNet Data Adapter Examples

 Report parameters “reportName” and “reportID”, and the return parameter (as “retval”), of ’CreateReport’: Class: Acme.Report.ReportGenerator Method: CreateReport Parameters: 0,reportName,reportID

 Report the first and third parameters of ’CreateReport’ as “reportName” and “reportID”: Class: Acme.Report.ReportGenerator Method: CreateReport Parameters: 1,3

 Report the first and third parameters of ’CreateReport’ as “NameOfReport” and “reportID”: Class: Acme.Report.ReportGenerator Method: CreateReport Parameters: 1:NameOfReport,3

 Report the first parameter of ’CreateReport’ as “NameOfReport”, the second parameter as “options”, the third parameter as “reportID”, and the return parameter as “retval”: Class: Acme.Report.ReportGenerator Method: CreateReport Parameters: 1:NameOfReport,*

 Report the parameter “reportID” for both ‘CreateReport’ and ‘DeleteReport’: Class: Acme.Report.ReportGenerator Method: * Parameters: reportID

 Report the first parameter of ‘CreateReport’, ‘DeleteReport’, and ‘PrintReport’, as “reportName”, “reportID”, and “reportName”, respectively: Class: Acme.Report.ReportGenerator Method: * Parameters: 1

JIDA Examples

 Report the first, third, and return parameters of ’CreateReport’ as “retval”, “p1”, and “p3”. The order of the integers that represent the position of the parameter in the method signature does not matter, so a Parameters value of 3,1,0 would have the same effect:

Aternity APM Version 11.4.2 65 Define a Configuration Screen

Class: Acme.Report.ReportGenerator Method: CreateReport Parameters: 0,1,3

 Report the first and third parameters of ’CreateReport’ as “p1” and “p3”, respectively: Class: Acme.Report.ReportGenerator Method: CreateReport Parameters: 1,3

 Report the first and third parameters of ’CreateReport’ as “reportName” and “reportID”, respectively: Class: Acme.Report.ReportGenerator Method: CreateReport Parameters: 1:reportName,3:reportID

 Report the first parameter of ’CreateReport as “NameOfReport”, and all other parameters (as “p2”, “p3”, and the return parameter as “retval”): Class: Acme.Report.ReportGenerator Method: CreateReport Parameters: 1:NameOfReport,*

 Report the first parameter of all methods (‘CreateReport’, ‘PrintReport, and ‘DeleteReport’’) as “p1”: Class: Acme.Report.ReportGenerator Method: * Parameters: 1

66 Aternity APM Version 11.4.2 Define a Configuration Screen

Examples: Member Fields of Complex Object Parameters

As described in “Syntax for Specifying Parameters“, collect filters can specify a reference to an object field of a parameter using the notation param.. This section shows examples of this syntax. The object fields and values are searchable and reported in the Call Details area of the transaction details window:

Classes Examples in this section are based on the following classes. Class TestObject has these fields: public class TestObject { String testStr; String name; int testInt; byte testByte; short testShort; long testLng; float testFlt; double testDbl; boolean testBool; char testChar; String[] strArr; Animal testAnimalObj; Cheetah childObj; ... }

Classes Animal, Cheetah, and Hat have these fields: public class Animal { public Animal(String classification, String name, int age) { ... } }

public class Cheetah extends Animal { Hat hat; public Cheetah(String classification, String name, int age, Hat hat) {

Aternity APM Version 11.4.2 67 Define a Configuration Screen

this.hat = hat; } } public class Hat { public Hat(String brand, String logo, String color, String brim) { ... } }

Class ComplexArgTests has the multObjFields method that has objects of type TestObject as parameters: public class ComplexArgTests { public void multObjFields(TestObject to1, TestObject to2) { ... }

Example Custom Filters The following examples show how different Parameters values in the “Edit Filter Dialog“ can report different fields from TestObject. The results show how the fields are reported (in the “Call Details“ section of the “Transaction Details“ window) with sample instance values passed in the multObjFields method. Report values of both parameters: Class: ComplexArgTests Method: multObjFields Parameters: 1,2 Results (these values are truncated as described in “Limits on Parameter Value Sizes in Traces“): p2=com.riverbed.test.feature.param.objects.complex.TestObject{testStr="test2",name="Stan",tes tInt=25,testByte=101,testShort=33,testLng=13,testFlt=6.03,testDbl=7.874,testBool=false,testCh ar=f,...}

p1=com.riverbed.test.feature.param.objects.complex.TestObject{testStr="test",name="Bob",testI nt=14,testByte=116,testShort=15,testLng=16,testFlt=3.14,testDbl=3.145,testBool=true,testChar= c,...} Report the value of field testStr in the to1 parameter: Class: ComplexArgTests Method: multObjFields Parameters: 1.testStr Results: p1.testStr=test Report the value of the Animal object in both parameters: Class: ComplexArgTests Method: multObjFields Parameters: 1.testAnimalObj,2.testAnimalObj Results: p2.testAnimalObj=com.riverbed.test.feature.param.objects.complex.Animal{classification="aves" ,name="Lance Armstrong",age=42}

p1.testAnimalObj=com.riverbed.test.feature.param.objects.complex.Animal{classification="mamma l",name="carl",age=12}

68 Aternity APM Version 11.4.2 Define a Configuration Screen

Report the value of the Animal name field in the to2 parameter: Class: ComplexArgTests Method: multObjFields Parameters: 2.testAnimalObj.name Results: p2.testAnimalObj.name=Lance Armstrong Report the value of the Cheetah object in the to2 parameter: Class: ComplexArgTests Method: multObjFields Parameters: 2.childObj Results: p2.childObj=com.riverbed.test.feature.param.objects.complex.Cheetah{hat=com.riverbed.test.fea ture.param.objects.complex.Hat{brand="47 Brand",logo="Reds",color="Red",brim="Flat"}classification="mammal",name="Hobie",age=4} Report the value of the Hat object in the to1 parameter: Class: ComplexArgTests Method: multObjFields Parameters: 1.childObj.hat Results: p1.childObj.hat=com.riverbed.test.feature.param.objects.complex.Hat{brand="New Era",logo="Red Sox",color="Dark Blue",brim="Curved"}

Example: Custom Stitching Using the > < Syntax

JIDA uses a variety of techniques to detect related cross-tier transactions. One technique involves injecting special SOAP headers that reliably identify cross-tier transactions for the web service implementations listed here. While APM can detect cross-tier transactions for other Web service implementations, it is most effective for web services where it injects the SOAP headers. JIDA injects the stitching SOAP headers for the following Web service implementations:

 Apache Axis (overview: http://axis.apache.org/axis/)

 Apache Axis2 Version 1.2 or later (overview: http://axis.apache.org/axis2/java/core/)

 JAX-RPC for WebLogic (overview: http://docs.oracle.com/cd/E12840_01/wls/docs103/webserv_rpc/intro.html)

 Metro (overview: http://www.oracle.com/technetwork/java/index-jsp-137004.html)

 Spring-WS 1.0 (overview: http://static.springsource.org/spring-ws/sites/1.0/)

 Spring-WS 2.0 (overview: http://static.springsource.org/spring-ws/sites/2.0/) This example describes a simple Java client-server application where the client has a SendMessage method and the server has a ReceiveMessage method with the following signatures: public void SendMessage(String message) public void ReceiveMessage(String message)

The client passes a unique value in the SendMessage method’s message parameter each time it calls the method. The server accepts that value as the ReceiveMessage message parameter value.

Aternity APM Version 11.4.2 69 Define a Configuration Screen

The > and < syntax (see “> | <“ in “Syntax for Specifying Parameters“ for more details.) at the end of a Parameters specification tells JIDA or the dotNet sub-agent to use the preceding parameter as the “stitching” value that APM uses to detect related transactions:

 The following filter reports the first parameter of the client SendMessage method parameter with a label of relationIdjava. It also specifies the > option so that JIDA supplies the parameter as the SENTREQUESTINJECTEDHEADER transaction trace file field:

Click Save to save the filter.

 Similarly, the following filter specifies the < option for the server ReceiveMessage method parameter. For the server, JIDA supplies the value in the RECEIVEDREQUESTINJECTEDHEADER field.

Click Save to save the filter and click Save at the bottom of the screen to save the configuration settings. APM uses those trace file fields is to identify the separate client and server transactions as related. Using this approach to override the automatically-supplied stitching value can enable the APM to detect related cross-tier transactions when it otherwise could not.

70 Aternity APM Version 11.4.2 Define a Configuration Screen

The following example shows the results of the successful custom stitching when the application runs. Here is the “Application Map Tab“ that shows that the transactions on the client (the Java_Jar_checksumClient_jar instance) and server tiers (Java_Jar_checksumServer_jar) are related:

The “Call Details“ of the “Transaction Details“ window shows that JIDA used the parameter name and value for the value of the SENTREQUESTINJECTEDHEADER and RECEIVEDREQUESTINJECTEDHEADER transaction trace fields:

Formatting of Different Parameter Data Types

 Primitive data types are reported as they would be formatted by default when written to the console.

 Strings are always enclosed in double quotation marks.

Aternity APM Version 11.4.2 71 Define a Configuration Screen

 Structures are reported as a string representation of their field names and values. For example: byteArray=[65,114,114,97,121] charArray=['A','r','r','a','y'] negInteger=-2147483648 specialChars=`~!@#$%^&*()-+=_/*<>"':;\.,{}[] 1500â•“1000 BCE retval={_items=["ArrayList: first String","ArrayList: second String","ArrayList: third String",null],_size=3,_version=3,_syncRoot=null}

Limits on Parameter Value Sizes in Traces

Both JIDA and the dotNet sub-agents limit the size of parameter values they will report. The sub-agents use different approaches but the approximate limits are as follows.

 Parameters are truncated at 500 characters.

 Arrays are truncated after 10 elements.

 Reporting includes at most one level of nesting. Deeper nesting is not visible. In addition: – Nested arrays are truncated after 5 elements. – Nested strings are truncated after 250 characters. Any time truncation occurs, the parameter values show an ellipsis (...). For nested arrays, the sub-agent indicate the number of omitted items. The following examples show elision indicators in bold: item={_items=[0,1,2,3,4,5,6,7,8,9,...1014 items omitted],_size=1000,_version=1000,_syncRoot=null,emptyArray=[],_defaultCapacity=4}

key="Day 14:Total # of characters=230: Quotes: A lie gets halfway around the world before the truth has ..."

list={m_Head={Key="Day 1:Total # of characters=230: Quotes: A lie gets halfway around the world before the truth has a...",Item=System.Collections.ArrayList,NextNode=com.riverbed.test.feature.param.collections.Link edList.Node`2}}

Optional Data to Report

Settings in this area control whether APM will monitor and collect data for various predefined categories of classes. As noted in this section, the monitored classes correspond to several of the categories described in the “Categories Reference“ topic.

Enabling Collection Requires Restarting Applications

When you change settings in this area (or in “Optional Data to Report (Java)“), APM changes the “Status“ column in the “Agent Details Screen“ for affected processes to instrument to May Require Restart. You only need to restart when you enable (check) an option. Disabling collection does not require restarting. See “Settings That Require Restarting Processes“ for more details.

Interaction of Optional Data Settings with Custom Filters

As described in “Monitoring Filters“, you can create custom filters that specify classes and methods of interest and the corresponding level of monitoring that you want.

72 Aternity APM Version 11.4.2 Define a Configuration Screen

Custom filters may specify classes and methods that overlap with those that correspond to Optional Data settings. For example, the System.Data.SqlClient.SqlCommand class is one of those that is monitored by enabling the “Database (JDBC and ADO)“ setting. In general, the custom filter takes precedence. For example:

 If the custom filter specifies Never in the “Collect (Interesting Only, Always, Never)“ setting but the Database Connectivity setting is selected, the custom filter takes precedence. In the following example, despite the fact that the Database Connectivity option is selected, APM will NOT monitor System.Data.SqlClient.SqlCommand methods, because the filter specifies Never. Other methods covered by the Database Connectivity setting will be monitored.

 In the following example, despite the fact that the Database Connectivity option is not selected, APM WILL monitor System.Data.SqlClient.SqlCommand methods, because the filter takes precedence. The methods will be reported under the “Application Code“ category described in the “Categories Reference“ topic. Other methods covered by the Database Connectivity setting will not be monitored.

Remote

Controls monitoring of socket-based network activity. The monitored classes correspond to the “Remote“ category described in the “Categories Reference“ topic.

Aternity APM Version 11.4.2 73 Define a Configuration Screen

Thread

Controls monitoring of thread activity in the .NET CLR (Common Language Runtime) or JVM (Java Virtual Machine) process. The “Threads Card“ in the “Instances Tab“ displays metrics controlled by this setting.

Garbage Collection

Controls monitoring of garbage collection data in the .NET CLR (Common Language Runtime) or JVM (Java Virtual Machine) process. The “Memory Card (Instances Tab)“ displays metrics controlled by this setting.

Web

Controls monitoring of .NET and Java web-tier classes. These classes correspond to the following categories described in the “Categories Reference“ topic:

 “Web“ (.NET)

 “JSP“ (Java)

 “Servlet“ (Java)

Web Services

Controls monitoring and identification of cross-tier activity among related web services. The monitored classes correspond to the “Web Service|Web Method“ category described in the “Categories Reference“ topic.

Message Bus (OSB and ASB)

Controls monitoring of the Oracle Service Bus . The monitored classes correspond to the “Message Bus“ category described in the “Categories Reference“ topic.

Database (JDBC and ADO)

Controls monitoring of database connections. The monitored classes correspond to the “Database (ADO)“ and “Database (JDBC)“ categories described in the “Categories Reference“ topic.

Log Packages

Adds log4net and log4j log messages to transaction trace data. Enabling this option causes APM to monitor the log4net and log4j classes and methods that include log messages as parameters. Like other method parameter values (see “Reporting Method Parameter Values“), you can search for the log messages in the “Search Tab“, and the log messages appear as parameter values in the “Call Tree Tab“ section of the “Transaction Details“ window. These classes are reported under the “Application Code“ category described in the “Categories Reference“ topic.

74 Aternity APM Version 11.4.2 Define a Configuration Screen

dotNet Sub-Agent For the dotNet sub-agent, the option monitors methods of the log4net.Core.LogImpl class. Each method corresponds to a logging level in the .NET application. Enabling Log Packages monitors the following methods and creates a corresponding parameter label that you can use with the “parametername“ search field:

Method Parmeter Label

Info infoMessage Warning warningMessage Error errorMessage Fatal fatalMessage

The following example search strings will return transactions containing log4net informational messages: parametername = infomessage parameter = infomessage* method = info

See https://logging.apache.org/log4net/release/sdk/html/T_log4net_Core_LogImpl.htm for more details on the log4net classes and methods.

JIDA Sub-Agent For the JIDA sub-agent, the option monitors the format method of the org.apache.log4j.PatternLayout class. The level of log messages reported depends on the logging level set by the application. For an application with a log level of INFO, this option will report INFO, WARNING, ERROR, and FATAL messages. DEBUG or TRACE messages are not reported unless the application is configured to that logging level. Enabling Log Packages creates the logMessage parameter label you can use in searches. The following example search strings will return transactions containing log4j debug messages: parametername = logmessage and parametervalue = *debug* parameter = logmessage* method = format

See https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html for details on the log4j class and method.

Searching for Log Messages In the Search tab, use auto complete to easily find messages of interest. (See “Auto Complete for Field Names, Operators, and Values“ for details.) For example, to search for a log4j debug message containing the string Successfully completed request, type this search string in the “Search Tab“: parametername = logmessage and parametervalue = successfully completed request

Auto complete narrows the possible options to matching results as you type:

Aternity APM Version 11.4.2 75 Define a Configuration Screen

Click Search to display matching transactions in the Search Results table. Click to open the “Transaction Details“ screen and navigate to the “Call Tree Tab“ section to see the log message:

SQL Bind Variables

Enables exposure of the names and values of “bind variables” in prepared SQL statements or stored procedures.

Note: Do Not Enable with Sensitive Data Use this option only when variables do not contain sensitive data. The variable names and values are not encrypted.

Unless this option is enabled, transaction trace data shows SQL statements with variable names but not values. For example, the following transaction trace shows the variable “studentname”, but it does not show the variable’s value: UPDATE scott.student SET address1=:address1, city=:city, zipcode=:zipcode,residenceState=:state WHERE studentName=:studentname

If you enable this option, the trace data will include the variable name (or position in the SQL statement), and its corresponding value. These values appear in the “Call Tree Tab“ section of the “Transaction Details“ window. The following example shows the contents of the SQL Bind Variable column, with bind variables and values for the previous SQL example:

76 Aternity APM Version 11.4.2 Define a Configuration Screen

@city:Nashua @zipcode:03063 @studentname:Charles Kao @address1:15 North Southwood Drive @state:NH

Optional Data to Report (Java)

Changing these settings affects monitoring of Java applications only (in other words, by the JIDA sub-agent).

Java Naming and Directory Interface (JNDI)

Controls monitoring of the Java Naming and Directory Interface (JNDI). The monitored classes correspond to the “JNDI“ category described in the “Categories Reference“ topic.

Enterprise JavaBeans (EJB)

Controls monitoring of JavaBeans. The monitored classes correspond to the “EJB, EJB Entity, EJB Message, EJB Session“ categories described in the “Categories Reference“ topic.

Java Message Service (JMS)

Controls monitoring of Java Message Service classes. JMS data provides information on Java Message Service (JMS) clients. When the Java Message Service configuration setting option is enabled, JIDA will generate transaction trace data and identify related cross-tier activity via JMS messages. The monitored classes correspond to the “JMS“ category described in the “Categories Reference“ topic. For information on supported versions of JMS, see the System Requirements document on the support site.

Remote Method Invocation (RMI)

Controls monitoring of remote method calls by instrumenting classes that implement the java.rmi.Remote interface. The monitored classes correspond to the “RMI“ category described in the “Categories Reference“ topic.

Portlets

Controls monitoring of Java portlets. The monitored classes correspond to the “Portlet“ category described in the “Categories Reference“ topic.

Aternity APM Version 11.4.2 77 Define a Configuration Screen

AppDomain Blacklist/Whitelist

These settings supplement the “Modify AppDomains Dialog“ in the “Agent Details Screen“ and provide more flexible configuration of which AppDomains you monitor. These settings apply only to .NET processes.

 Choose Exclude or Include to specify a “blacklist” or “whitelist”, respectively, of AppDomains.

 Specify a comma-separated list of AppDomain names. Names are not case sensitive. You can use an asterisk (*) as a wildcard character anywhere in each name to match zero or more characters. APM discovers new AppDomains as they start. For processes assigned to this configuration, it compares AppDomain names with the list. If the name matches an Exclude entry, it will not be instrumented. If it matches an Include entry, it will. The AppDomains appear in the “Modify AppDomains Dialog“ with the appropriate setting. The default for these settings is an empty Exclude list. This results in automatically instrumenting all AppDomains as they are discovered (provided the process itself is instrumented). Use this setting when you want to control instrumentation of AppDomains that are not currently running:

 Some .NET environments automatically create many transient AppDomains that are not interesting to monitor (and which would cause excessive overhead if they were instrumented). In this case, specify Exclude and values that will match the unwanted AppDomains. When the matching AppDomains run, they appear in the “Modify AppDomains Dialog“ but with the Include? setting not checked. (If the AppDomain matches a wildcarded string, the Include? setting is not checked and disabled.)

 You may want to suppress automatic instrumentation and monitor only a known set of AppDomains. In this case, specify Include and the names of the AppDomains you are interested in. Only matching AppDomains will appear in the “Modify AppDomains Dialog“ with the Include? setting checked. (If the AppDomain matches a wildcarded string, the Include? setting is checked and disabled.) Keep in mind that changes you make in these settings may affect the Include? setting in the “Modify AppDomains Dialog“:

Any Exclude entry that matches a name in the dialog list will cause the Include? setting for the matching AppDomain to be unchecked (and disabled if it matches a wildcarded Exclude entry). Similarly, any AppDomain that does NOT match an Include entry will have its check box unchecked (and possibly disabled).

78 Aternity APM Version 11.4.2 Define a Configuration Screen

Logging

These settings control the level of detail in log files for dotNet and JIDA sub-agents. Agent and sub-agent log files reside in the /Panorama/hedzup/mn/log directory on the agent system.

 Log file names for the dotNet sub-agent start with the prefix DA-dotNet.

 Log file names for the JIDA sub-agent start with the prefix DA-JIDA.

Accessing Log Files

You can get access to log files in any of the following ways:

 Use a web browser. The agent serves log files at this URL: http://:2111/log/

 Create and download a “diagnostic bundle” that includes the log files. On the “Agent Details Screen“, click the “Download the agent logs and diagnostic files“ link.

 Edit the log files directly in the /Panorama/hedzup/mn/log directory.

Different Files for Sub-Agent Activity and Lists of Methods

The sub-agents writes different details to separate log files:

 Messages logging sub-agent activity are written to .txt files. These file names include the name the sub-agent assigns to the processes to instrument, the time the sub-agent started, and the process identifier. New versions of these activity log files are created each time the application starts. For example:

 A list of application methods that the sub-agent instrumented and is monitoring are written to _methods.txt files. (If the “Include all discovered methods (Restart Required)“ option is selected, this list includes all methods that the sub-agent discovered.) For example: DA-dotNet_IIS_DefaultAppPool_methods.txt DA-JIDA_prunsrv_methods.txt These files are overwritten each time the application starts. To limit their size, the sub-agents stop writing to them after 1 million messages.

Enable Verbose logging

Writes additional informational messages to the sub-agent activity log. When you enable verbose logging, the dotNet or JIDA sub-agent write a message confirming the change similar to the following: 05/21/2015 03:31:19 PM, dotNet , 6516 , INFO , Configuration change: logging.verbose = true

Aternity APM Version 11.4.2 79 Define a Configuration Screen

You do not need to restart the application for verbose logging setting to take effect. However, many messages are written out only when application starts, so if you want to see them, you do need to restart.

Note: This setting causes high disk activity and rapidly results in very large log files. Leave it disabled for lowest overhead.

Include application exceptions

Writes any exceptions from the .NET or Java application to the corresponding sub-agent activity log file. This option is useful primarily to diagnose startup problems with the sub-agent.

Note: This setting slows application performance, causes high disk activity and rapidly results in very large log files. Use this option only at the direction of Aternity support.

When you enable this option, the dotNet or JIDA sub-agent write a message confirming the change similar to the following: 05/22/2015 12:55:07 PM, dotNet , 7948 , INFO , Configuration change: application.exception.logging.report = true

05/22/2015 12:06:37 PM, JIDA , 3132 , INFO , Application exception logging changed to enabled

The sub-agent writes the Logging Application Exception informational message before each application exception in the log. For example: 05/22/2015 01:04:06 PM, dotNet , 7704 , INFO , Logging Application Ex ception: <== System.Data.SqlClient.SqlException in thread 39820352 (00001e18) app domain /LM/W3SVC/1/ROOT/DB Query-1-130767728543866252 message: hresult: 00000000 at: System.Data.SqlClient.SqlConnection::OnError + 0xe0 System.Data.SqlClient.TdsParser::ThrowExceptionAndWarning + 0xf5 System.Data.SqlClient.TdsParserStateObject::WriteSni + 0x16c . . . ==>

Include all discovered methods (Restart Required)

Note: Restart Processes After Changing this Setting Changing this setting changes the “Status“ column in the “Agent Details Screen“ for affected processes to May Require Restart, but always requires restarting the processes assigned to this configuration.

This setting writes additional messages to the DA-dotNet__methods.txt and DA-JIDA__methods.txt files.

80 Aternity APM Version 11.4.2 Define a Configuration Screen

By default, these log files always include messages for every method that the sub-agent instrumented and is monitoring. When you enable Include all discovered methods, these log files also include messages for every method that the sub-agent discovered but that it did not instrument. See “Reviewing Log Files to See Classes and Methods that Are, Are Not Instrumented“ for more details on the messages.

UNIX File Creation Permissions

These settings apply only to the JIDA sub-agent on Unix-like operating systems. They control the Unix file permissions for transaction trace files and their parent directories. When the JIDA sub-agent monitors a Java process, the user account that starts the Java process being monitored creates and owns the following files:

 The continuous/ directory in the /Panorama/hedzup/mn/userdata/captures/ directory.

 The subdirectories and transaction trace files below the continuous/ directory. (The same user also creates the DA-JIDA log files in the /Panorama/hedzup/mn/log/ directory, but permissions on those files do not typically cause problems and are not affected by the settings described here.)

Group

By default, the directories and files are created with group permissions. This default excludes access to any account that is not in the same group as the user account that starts the Java process that JIDA is monitoring. Other users can not read the files. This default behavior will cause problems if the APM administrative account is not in the same group as the user that starts the Java process that JIDA is monitoring. In this case, the agent (the dsa process) can not read the trace files and they are never copied to the analysis server.

Others

Choose Others to relax the permissions so that any user can read the directories and transaction trace files. This is less secure but avoids the issue of the dsa process not being able to read trace files. If you change the setting to Others from Group before starting any Java processes with JIDA monitoring, all directories and trace files will be created with read permissions for others. However, if you change the setting after starting a Java process with JIDA, the continuous/ directory and its children will already have been created with the more restrictive read permission for group members. In this case, APM will attempt to change the permissions on the subdirectories and trace files, but may not succeed. If this happens, JIDA writes an error to its log file and you will have to manually change the permissions.

Follow umask Setting

Choose Follow umask Setting to stop JIDA from applying any permissions to the transaction trace files and parent directories. Instead, it relies on the umask setting for the user that started the Java application. The user file-creation mode mask (umask) determines the default file permission for new files and directories. See the following link for more details: http://en.wikipedia.org/wiki/Umask

Aternity APM Version 11.4.2 81 Define a Configuration Screen

Transaction Tracing

These settings control various aspects of transaction trace monitoring. Here, the term "transaction" refers to a stream of calls within a single thread of execution. The first transaction in a trace file starts with the outermost method call that JIDA or the dotNet sub-agent detects after tracing begins. It ends with the matching completion for that call. Subsequent tracing transactions start with the outermost call in the thread and end with the matching completion.

HTTP Cookies/Headers

These settings specify the names of HTTP headers and cookies that JIDA or the dotNet sub-agents will include in transaction traces. By default, these settings are empty, and the sub-agents do not write any header and cookie data. This data can be verbose and add substantially to the size of transaction trace files and the disk activity to write them. Specify a comma-separated list of cookie or header names whose data you want to include in traces. Use an asterisk (*) as a wildcard character to match one or more characters in a name or by itself to match all names. For example: HTTP Cookies *SessionId HTTP Headers Host,Accept* The sub-agents compare the lists with header and cookie names in HTTP pages they are monitoring. For matching names, they write the names and values to the trace file. To see what cookies and headers are available, use browser tools to examine pages directly. Alternatively, specify * for both settings to see all cookies and headers in traces, then refine the setting to include only those of interest.

Enable End-User Experience Tracing Cookie (legacy) This option controls whether JIDA and the dotNet sub-agents generate an HTTP cookie (_op_aixPageId) containing an identifier that the analysis server uses to link web page performance data with related application transactions. Typically, enabling this option is unnecessary. When you enable the “Collect End-User Experience Data“ option to automatically add the JavaScript snippet to instrumented web pages, the sub-agents include a variable that serves the same purpose as the tracing cookie. This specialized setting is useful only in cases where the sub-agents cannot add the JavaScript snippet (for example, if the HTML content is compressed). In such cases, the snippet must be added to pages in some other manner and this setting enabled. However, note that the cookie may be identified as a risk in security scans (it does not set either the Secure or HttpOnly flags). Because of this, enable this option only when Collect End-User Experience Data fails to add the JavaScript snippet.

Parameter Reporting

Check boxes in this area enable or disable parameter reporting for specific categories of classes. Enabling a setting automatically reports parameter values for the corresponding category of classes. When enabled, parameters appear in the analysis server interface in the same manner as those explicitly specified in custom filters (see “How Parameters Appear in the User Interface“).

82 Aternity APM Version 11.4.2 Define a Configuration Screen

Report Web Tier Parameters Enables or disables parameter reporting for web tier classes. This setting affects the “Web“ and “Web Services“ categories of classes. The parameters reported include parameters from submitted HTML forms, for both GET and POST methods. The following diagram shows how form parameters from a .NET web application are exposed in the user interface:

For the Report Web Tier Parameters option only, there is a Parameter blacklist text box. This area accepts a "blacklist" of web parameter names whose values contain sensitive data. Values of parameters specified here will be suppressed in transaction traces. The parameter names are still included in the trace data and appear as usual in the APM interface, but the values are replaced by the string blacklisted_in_config. Note that the blacklist only affects parameter reporting for web tier classes. It has no effect on parameter reporting enabled via custom filters as described in “Reporting Method Parameter Values“. Add names of parameters whose values you want to suppress:

 Separate names with commas.

 Values are not case sensitive (password is the same as PASSWORD).

 Use * to match one or more characters (*pass*).

 The list cannot exceed 257 characters.

Aternity APM Version 11.4.2 83 Define a Configuration Screen

In addition to any parameter names specified here, there is a default blacklist of parameters names that are likely to contain sensitive data (for example, *pass*). Both JIDA and the dotNet sub-agents automatically suppress reporting of values for these parameters, even if the Parameter blacklist text box is empty. The Parameter blacklist text box is used to add to this default list.

Report EJB Parameters Enables or disables parameter reporting for “Enterprise JavaBeans (EJB)“ classes in Java applications.

Report RMI Parameters Enables or disables parameter reporting for “Remote Method Invocation (RMI)“ classes in Java applications.

Transaction Settings

Maximum string length This setting limits the size of strings that sub-agents allow in transaction trace data. Strings longer than this value will be truncated. Strings whose lengths are affected by this setting include:

 SQL statements, which limits the size of values returned by the “sql“ search field in the “Search Tab“.

 URL and exception strings (see the “url“ and “exception“ search fields).

 Method parameters (see “Reporting Method Parameter Values in Transaction Traces“).

 SENTREQUESTINJECTEDHEADER and RECEIVEDREQUESTINJECTEDHEADER trace file fields used to identify related transactions (see “Example: Custom Stitching Using the > < Syntax“). The minimum allowed value is 200 characters. The default value varies depending on the preset level (see “Data Collection Settings: Preset Levels“). Increasing the value potentially exposes more detail but increases the overhead to generate transaction trace data.

Maximum call duration This setting specifies the longest call sequence (in minutes) that APM will consider a single transaction when generating transaction trace data. This limit allows APM components to handle long-running transactions. In most cases, you should not have to change this setting. In general, JIDA and the dotNet sub-agents determine the start and end of transactions by monitoring the method call stack. A transaction starts with the outermost method call that JIDA or the dotNet sub-agent detects after tracing begins. It ends with the matching completion for that call. In between, the call stack increases and decreases in depth as methods call other methods and return.

84 Aternity APM Version 11.4.2 Define a Configuration Screen

The following figure shows a simple case where the call stack depth increases to 3:

When the call stack depth returns to zero, the sub-agents mark the transaction as complete. The next method call begins a new transaction, and the APM can store the data about the completed transaction. This technique works in most cases, but not when transactions run indefinitely. Examples where transactions can run indefinitely include: Thread pools: The application uses thread pools that implement a method that runs for the lifetime of the application. If the sub-agents monitor such a method, they will never detect the end of any transaction it is involved in.

Aternity APM Version 11.4.2 85 Define a Configuration Screen

Stuck calls: Because of a problem, a method call never completes. In this "stuck call" case, the call stack depth increases to a certain point and never decreases.

The sub-agents use the value of the Maximum call duration setting to handle such cases:

 In the thread-pool case, once the maximum duration elapses, the sub-agents adjust the call-tree depth to zero. This ends the transaction and effectively excludes the thread-pool method from the call hierarchy.

 In the stuck-call case, the sub-agents include a "did not finish" indicator in the trace data along with detail about the stuck method and how long it was stuck.

Disk Limits

These settings control disk usage and activity by transaction trace files.

Maximum disk used Maximum disk used limits the disk space that trace files for a single monitored process can use. If trace files for any single process exceeds Maximum disk used, APM deletes the oldest files in the subdirectory for the process on the agent system (\Panorama\Hedzup\mn\userdata\captures\continuous\).

Note: Total Disk Usage May Be Higher Because Maximum disk used applies separately to the subdirectory for each monitored process, the combined disk usage by trace files for multiple processes will be higher. For processes controlled by the same configuration, the highest possible total usage will be the Maximum disk used value multiplied by the number of processes.

Note that trace files remain on the agent even after they are copied by the analysis server. Trace files accumulate and consume disk space until their subdirectory exceeds the Maximum disk used setting.

Minimum disk free Unlike Maximum disk used, Minimum disk free applies to the entire disk where the agent software is installed. If the available disk space falls below Minimum disk free, APM stops tracing for the processes. To resume tracing, you must change the setting or delete files so the available space complies with the setting.

86 Aternity APM Version 11.4.2 Define a Configuration Screen

Maximum write rate Limits the rate at which APM will write data to trace files. This setting limits the processing load imposed by the tasks on the agent system. Similarly, it also limits the network load when the analysis server copies trace files. APM uses two techniques to limit trace disk throughput:

 Exclude short method calls

 Omit less-interesting fields for each transaction If the data rate exceeds the limit specified by this setting, APM omits the shortest method calls (1 millisecond or less) and least-interesting fields (initially, bytes sent and received) from the data it writes. If the rate continues to exceed the limit, APM omits longer method calls and additional fields until the rate complies with the specified limit.

Configuration Overrides

Do not use this setting unless directed to do so by Aternity technical support. It accepts internal JSON properties and values that will be sent to the processes assigned to this configuration. APM checks for valid JSON in this field and disables the Save button if the JSON is not valid. However, there is no other validation. Supplying invalid property names, values or format will cause unpredictable results. Also, configuration overrides take precedence over any settings that conflict with them. Once a configuration includes any configuration overrides, the Overrides Enabled banner appears in the “Current Configuration“ area to warn of this possibility.

Save / Revert

Click Save to save changes and open the “Agent List Screen“. You must click Save to save changes. Navigating from the screen by clicking another menu option will discard any changes you have made. Click Revert to discard changes and remain on this screen.

Aternity APM Version 11.4.2 87 Define a Configuration Screen

88 Aternity APM Version 11.4.2 CHAPTER 8 JavaScript Snippet

When you select the “Collect End-User Experience Data“ option in the “Define a Configuration Screen“, APM generates a JavaScript snippet that it uses to monitor web page performance from the perspective of end users.

Customize Snippet

When you click the “Customize Snippet“ link in the “Define a Configuration Screen“, the screen displays tabs with specialized settings for customizing the JavaScript snippet. Typically, you do not need to change these settings.

Library Location

This option changes the server that hosts a JavaScript instrumentation library required by APM to monitor web page performance.

 Served from the cloud This option serves the library from the cloud (an Akamai content delivery network (CDN) server). This option is fast and fault tolerant, and keeps the footprint as light as possible. The library is automatically updated as necessary.

 Self-served This option serves the library from a web server you specify, typically the same server that hosts the web application you want to monitor. Hosting the library in the same application server eliminates a dependency on another server. When you select this option, configuration instructions for downloading and hosting the library appear in the dialog window. Copy the downloaded library from the browser window where it downloaded and write it to a file that is served by your web server. Then specify the served file location in the Location setting.

Aternity APM Version 11.4.2 89 JavaScript Snippet

If you enable collection of “AJAX“ data, be sure to download the corresponding library.

Page Tags

Page tags are optional variables in the JavaScript snippet that you can use to collect additional data available from web pages being monitored. This data is collected in addition to the web page performance data that APM collects automatically. Page tags are exposed in the analysis server web interface as follows:

 As the “Page Tags“ transaction type definition criterion in the “Define a Transaction Type Screen“

 As the “pageload.time.backend“ and “pagetag.1, pagetag.2, and pagetag.3“ search fields values in the “Search Tab“

 In the “Page Tag 1 Page Tag 2 Page Tag 3“ columns of the “Search Results“ table in the “Search Tab“

 In the “Page Tag “ detail in the “Overview Tab“ section of the “Transaction Details“ screen To enable page tags, select the Add page tags to snippet with the following values option. You can hard-code page tag values in the Page Tags area (see “Hard-Coding Page Tag Values“) or use scripting (see “Setting Page Tag Values in Web Page Scripts“). In either case, values shown in the interface are truncated at 50 characters.

Hard-Coding Page Tag Values

To add hard-coded page tag values to the JavaScript snippet, first select the Add page tags to snippet with the following values option. Supply a value in at least one of the Page Tag x settings as a character string enclosed in single or double quotation marks. For example: "values apply to all processes in the configuration"

'DB query ASP page'

After you provide a value, click View Snippet to verify that values were added as expected. Click Save to save the configuration. For example: . . .

To confirm the variable values are propagating as expected, exercise the application. The browser’s View Page Source feature should show the snippet with the variables set immediately following: . . .

Add the Page Tag 1 and Page 2 columns to the “Search Results“ table in the “Search Tab“:

Aternity APM Version 11.4.2 91 JavaScript Snippet

After a few minutes, the results table should show the expected values:

Data Collection

To avoid exposing potentially sensitive data, you can prevent APM from collecting certain information. Clear the options for the data you do not want to collect:

 Page title (text defined in the page’s title tag)

 URL query string (all text following, and including, the first ? character)

 URL fragment (all text following, and including, the first # character) Note that changing these options might affect any previously defined “Transaction Types“ and the order that they are matched.

User Tracking

Disable user tracking by cookie

By default, APM anonymously tracks users (within the instrumented site only) by placing a cookie on the user’s machine. Clear the setting to turn off this behavior. Even if you disable user tracking by cookie, APM still collects the IP address for each page view. APM uses IP address to determine the geographical location, company name, and ISP.

Add a username variable to snippet

If web application pages have access to the name of the browser user, you can configure the JavaScript snippet to collect the user name along with web page performance data. If configured, the value is exposed in the analysis server web interface:

 As the “username“ search field value in the “Search Tab“

 In the User column in the search results table (if enabled)

 In the “Overview Tab“ section of the “Transaction Details“ screen Select the Add username variable... option to hard-code the user name. Supply a hard-coded string enclosed in single or double quotation marks. For example: 'john.doe'

"jane.doe"

92 Aternity APM Version 11.4.2 JavaScript Snippet

After you provide a value, click View Snippet to verify that the username value was added as expected. Click Save to save the configuration. You can also use scripting to assign a value to the special RVBD_EUE.username variable. This is the same approach as described for page tags in “Setting Page Tag Values in Web Page Scripts“.

AJAX

This option enables collection of data for AJAX (Asynchronous JavaScript and XML) transactions. AJAX refers to a group of related web technologies that applications use to communicates with a server in the background, without interfering with the current state of the page. This option is disabled by default. Enabling AJAX transactions increases the volume of web page performance data. By default, when you enable collection of AJAX data, you also enable the following options:

 Extend page load time to include initial AJAX transactions An “initial AJAX transaction” is a specially-calculated delay component that represents the impact of AJAX transactions on the response time of a page view. Typically, such transactions must complete before the user can interact with the page. APM detects initial AJAX transactions based on proprietary algorithms. This option adds the time for initial AJAX transactions to the calculation of page load time.

 Synchronously load JavaScript library for AJAX pages (Strongly recommended) For non-AJAX transactions, APM loads the JavaScript library asynchronously to avoid any delay to web pages. However, to capture all AJAX transactions (specifically, early calls before the JavaScript snippet loads), APM must load the library synchronously. Leave this option selected to capture all AJAX transactions. If you customized the “Library Location“ to use the “Self-served“ option, make sure that the self-served location serves the JavaScript library with AJAX support.

Advanced Options

Use this tab at the direction of Aternity support to add one or more variable names and values to the JavaScript snippet. Supply variable names and values as a comma-separated list. Enclose string values in single or double quotes: name1 : "string1", name2 : 1234 , name3 : true

Click View Snippet to verify that the variables were added as expected. Click Save to save the configuration.

Aternity APM Version 11.4.2 93 JavaScript Snippet

94 Aternity APM Version 11.4.2 CHAPTER 9 Agent and Configuration History Screen

This screen lists a variety of events relating to agent configuration. You can filter the events displayed by choosing criteria in the Description and Status lists. Events include the following:

Event Description

Created a new configuration Changes to configurations that control what application data that JIDA and the Modified a configuration dotNet sub-agents collect on agents. Any time you create, modify, or delete a Deleted a configuration configuration in the “Configurations Screen“ or “Define a Configuration Screen“, APM creates a corresponding entry in this screen. These entries show the configuration name in the Source column. Click the name to open the “Define a Configuration Screen“. Discovered a new Agent Connection by a new agent to the analysis server. (The new agent appears in the “Agent List Screen“.) These entries show the agent name in the Source column. Click the name to open the “Agent Details Screen“. Discovered new processes Discovery by an agent of new processes to instrument. (The processes appear to instrument in the Agent Details screen.) These entries show the related agent name in the Source column. Click the name to open the “Agent Details Screen“. Removed processes to Removal of processes to instrument (using the “Delete (X)“ option in the instrument Processes to Instrument list in the Agent Details screen).

Instrumented processes Instrumentation and “uninstrumentation” of processes to instrument (using Uninstrumented processes the “Instrument Option (Restart Required)“ in the Processes to Instrument list in the Agent Details screen).

Configuration and instrumentation changes require the analysis server to send changed configuration options to the agent. If any of the affected processes do not receive the changes, the status appears as Failed or x Failures. Click the status value to open a dialog to re-try the operation. Note that the history is limited to the last 15,000 events. Once the number of events reaches 15,000, the analysis server deletes the oldest 5,000.

Aternity APM Version 11.4.2 95 Agent and Configuration History Screen

96 Aternity APM Version 11.4.2 CHAPTER 10 Alert History Screen

This screens lists alerts triggered when metric values or transaction types violate alert conditions. (Create and modify alerts in the “Alert Definitions Screen“.)

Active Whether an alert is active. The presence of the icon indicates that the metric value or transaction type continues to violate the alert conditions specified in the “Define an Alert Screen“. The alert clears (and the icon disappears) after the metric value or transaction type complies with the alert conditions for 60 seconds.

Name The name of the alert, as specified in the “Define an Alert Screen“.

Triggered Time that the alert was triggered.

Cleared If the alert is not active, the time it cleared.

Type Type of threshold violation that triggered the alert: • “Environmental Alerts“ are based on “Environmental Thresholds“ violations. • “Transactional Alerts“ are based on evaluating the count of major and minor “Transactional Thresholds“ violations. Source For environmental thresholds, the name of the server where the alert occurred. Click the name to jump to the “Servers Tab“ with the server highlighted. For transactional thresholds, the name of the transaction type whose thresholds were violated. Click the name to jump to the “Transaction Types Tab“ with the transaction type highlighted. Metric Name of the metric

Value Value of the metric that triggered the alert.

Violation % For transactional alerts only, the percentage of transactions that were in violation when the alert was triggered.

Apdex For transactional alerts only, the Apdex score when the alert was triggered. See “Transactional Thresholds and Apdex Scores“ for details on how the score is calculated.

Aternity APM Version 11.4.2 97 Alert History Screen

98 Aternity APM Version 11.4.2 CHAPTER 11 Alert Definitions Screen

This screen lists any alerts that have been defined and options to create and modify alert definitions. If no alerts have been defined, the Alert Definitions table is empty. Alert definitions specify conditions that control whether threshold violations will be promoted to an alert. (Thresholds are separately configured, in the “Threshold Overview Screen“.) If the conditions in the alert definition are met, APM adds an entry to the “Alert History Screen“ and, if configured, sends notifications as described in the “Define an Alert Screen“ topic. APM evaluates alert conditions when any threshold violation occurs. The screen has the following controls:

 The ON | OFF Alerts setting toggles between enabling and disabling all alerts. Toggling alerts OFF implicitly clears any alerts that are currently in violation. You can also enable and disable individual alerts in the “Define an Alert Screen“, as indicated in the Enabled column of the table.

 To add a new alert definition, click Define an Alert. The “Define an Alert Screen“ opens.

 To edit an existing alert definition, click the edit icon ( ) in the row for the alert definition you want to edit. The “Define an Alert Screen“ opens with the settings for that alert definition.

 To delete an existing alert definition, click the delete icon ( ) in the row for the alert definition you want to delete.

 To sort the list, click on any column heading to sort the column in ascending order based on the column value. Click the column heading again to sort in descending order. An icon indicates ascending or descending sort order.

Aternity APM Version 11.4.2 99 Alert Definitions Screen

100 Aternity APM Version 11.4.2 CHAPTER 12 Define an Alert Screen

Overview

This screen opens when you add or edit an alert definition from the “Alert Definitions Screen“. Alert definitions specify conditions that control whether threshold violations will trigger an alert. (Thresholds are separately configured, in the “Threshold Overview Screen“.) If the alert is enabled and the conditions in the alert definition are met, APM:

 Records the alert in the “Alert History Screen“.

 Potentially sends an alert notification: – As an email message to addresses specified in the “Email Settings“ area of this screen – As an SNMP trap to an SNMP receiver, if enabled in the “SNMP Settings“ area of this screen – As a ServiceNow incident alert, if enabled in the “ServiceNow Settings“ area of this screen APM evaluates alert trigger conditions when any threshold violation occurs. Create alert definitions that combine different trigger conditions with different thresholds to generate alerts when you want.

Summary of Steps to Define an Alert

Use this screen to define an alert:

 Provide a name and enable the alert in the “General“ area.

 In the “Alert Source“ area, choose the type of alert and one or more thresholds that will be evaluated against the trigger conditions.

 Specify trigger conditions that control whether the associated threshold violations will trigger an alert. There are different settings for “Environmental Alerts“ (based on environmental thresholds) and “Transactional Alerts“ (based on transactional thresholds).

Aternity APM Version 11.4.2 101 Define an Alert Screen

 Enable alert notification destinations as desired: –In “Email Settings“, specify recipients for the alert email and other settings. –In “SNMP Settings“, enable sending SNMP traps as an alert notification. –In “ServiceNow Settings“, enable sending alert notifications a ServiceNow incident alert.

 “Scheduling“ settings are optional and specify times you do not want threshold violations to trigger the alert.

How APM Evaluates Alert Conditions

APM evaluates conditions specified every minute and triggers an alert when the conditions are met. Alerts clear after the first minute that conditions are no longer met. The following diagram shows an example of settings specified for an environmental alert and how values for the corresponding metric will trigger and clear alerts.

102 Aternity APM Version 11.4.2 Define an Alert Screen

Default Alert Definitions

The analysis server installation creates the following alert definitions:

 Default Transactional Alert Definition

 Default Environmental Alert Definition In upgrades from versions earlier than Version 10.5, these alert definitions are enabled and preserve settings migrated from the old Alert Settings screen. In those earlier releases, there was a single set of transactional and environmental alert settings. In new installations, these alert definitions are not enabled. You can delete them or adapt them to suit your purposes.

General

Provide a name and optional description for the alert. By default, the alert is enabled. Clear the Enabled check box to disable this alert.

Alert Source

Alerts are triggered by either an environmental or transactional threshold (not both). Choose the type of threshold whose violations APM will evaluate to potentially trigger alerts:

 Choose Environmental to display a list of environmental thresholds that have been defined. This list is the same as shown in the Environmental tab of the “Threshold Overview Screen“. When you choose this option, the “Environmental Alerts“ area appears.

 Choose Transactional to display a list of transactional thresholds that have been defined. This list is the same as shown in the Transactional tab of the “Threshold Overview Screen“. When you choose this option, the “Transactional Alerts“ area appears. Select the check boxes for any combination of available thresholds that you want APM to evaluate to determine whether to trigger alerts.

Environmental Alerts

This area appears when you choose Environmental as the “Alert Source“. Environmental alerts are based on environmental threshold violations. These thresholds are configured in the “Environmental Thresholds“ dialog (accessed from the “Threshold Overview Screen“). For environmental alerts, there is only one trigger condition. It specifies the number of minutes over which any of the thresholds selected in the “Alert Source“ area must persist before APM generates an alert.

Transactional Alerts

This area appears when you choose Transactional as the “Alert Source“.

Aternity APM Version 11.4.2 103 Define an Alert Screen

Transactional alerts are based on transactional threshold violations. These thresholds are configured in the “Transactional Thresholds“ dialog (accessed from the “Threshold Overview Screen“) and are themselves based on transactions that match a specific transaction type. For transactional alerts, there are multiple trigger conditions you can configure. There are two sections, with conditions that apply only to specific threshold metrics:

 Conditions under Trigger alerts for response time based thresholds apply only to thresholds that specify time-based metrics.

 Conditions under Trigger alerts for count time based thresholds apply only to thresholds that specify count metrics. The following figure shows which trigger conditions apply for different threshold metrics.

Response-Time Threshold Trigger Conditions

These trigger conditions apply only to transactional thresholds that specify time-based metrics. They provide flexibility to generate alerts for behavior that represents a problem without triggering “alert storms” of unnecessary warnings.

 More that X% of transactions are violating The percentage of all transactions that match the transaction types corresponding to the thresholds specified in the “Alert Source“that must be in violation to potentially generate an alert. Lower values lead to more alerts. 1 is the lowest value allowed.

104 Aternity APM Version 11.4.2 Define an Alert Screen

 for at least Y consecutive minutes The interval over which the percentage of transactions violating must persist. Set to 1 to effectively disable this condition.

 and with at least Z transactions each minute The minimum number of transactions that must meet preceding conditions. See “Transactional Alert Example“ for an example of how the trigger conditions interact with each other and with threshold and transaction type definitions.

Count Threshold Trigger Conditions

This trigger condition applies only to transactional thresholds that specify count metrics. It specifies the number of minutes over which any of the thresholds selected in the “Alert Source“ area must persist before APM generates an alert.

Aternity APM Version 11.4.2 105 Define an Alert Screen

Transactional Alert Example

To understand how APM determines when to send a transactional alert, consider the following transaction type, threshold and alert configuration:

For this example, suppose this is the only transaction type, and that there are 100 matching transactions every minute, as follows:

 51 take .5 seconds

 21 take 1 second

 The remainder take .2 seconds This behavior will generate 51 minor threshold violations and 21 major violations every minute and meets the 25% violation criteria in the alert configuration. However, as specified in the consecutive minutes criterion, APM will not generate an alert until after 5 minutes. Also, the alert will not clear until BOTH major and minor violations drop below 25% of transactions for at least one minute.

106 Aternity APM Version 11.4.2 Define an Alert Screen

Email Settings

These settings configure email alert notifications:

 Who receives the email alert message

 The format of the message

 Whether to send a message when an alert condition clears Customize the content of the email in the “Alert Notification Template Configuration Screen“.

Recipients

Specify recipients as a comma-separated list of email addresses in the text box.

Message Format

Choose either Human Readable or JSON.

Human Readable

You can customize the subject and body of this format in the “Alert Notification Template Configuration Screen“. This example shows a sample using the default email template: From: [email protected] Sent: Tuesday, September 13, 2016 4:29 PM To: test Subject: ALERT: AppInternals - component CTSecure (beth01.tfnportal) disk2 CPU Usage >=50 %

Attention, an alert has been triggered at 2016/07/25 18:06:41 UTC for component CTSecure (beth01.tfnportal) disk2, which is violating a CPU Usage threshold.

Visit the alert page for more details. * Current CPU Usage: 89 % (major violation) * Violations started at 2016/07/25 17:46:41 UTC (20 seconds) * Average CPU Usage over violation period: 30 % Alert Details Name: CPU Usage Description: Monitors the CPU usage and will alert if going outside of the set thresholds Configured thresholds: * Minor: <10 percent; Major: <5 percent * Minor: >50 percent; Major: >80 percent Server Details Server Name: beth01.tfnportal Server Tags: application:Tradefast, deployment_group:Infrastructure, instance:Web Server

JSON

The JSON (JavaScript Object Notation) format is useful for sending notification to an automated system for parsing, such as to generate a troubleshooting ticket). The JSON format contains details on the alert. Unlike the “Human Readable“ format, you cannot use the “Alert Notification Template Configuration Screen“ to customize the details of the body of the email in the JSON format (you can customize the email subject).

Aternity APM Version 11.4.2 107 Define an Alert Screen

This example shows an excerpt of the JSON email format for an alert-cleared email for an environmental alert: { "alert": { "above_threshold": true, "active": false, "average_value": 30, "blackout": { "friday": true, "from_seconds": 0, "monday": true, "saturday": false, "sunday": false, "thursday": false, "timezone": "UTC", "to_seconds": 3600, "tuesday": false, "wednesday": true }, "clear": { "average_value": 28, "current_value": 30, "time": 1469470135.0 }, "current_value": { "above_or_below_threshold": ">=", "above_threshold": true, "severity": "major", "threshold": 50, "value": "89" }, . . .

Miscellaneous

Send Email once alert is cleared

Select this setting to send a second notification once the alert clears.

Send Test Email

Use the Send Test Email button to send a test message to the addresses in the “Recipients“ list. Note that the test email uses hard-coded sample values for the tags specified in the template. Values for the alert name or description tags, for instance, will not reflect the actual alert.

SNMP Settings

This area contains a single setting that enables sending an SNMP trap when threshold violations trigger an alert.

108 Aternity APM Version 11.4.2 Define an Alert Screen

The Simple Network Management Protocol (SNMP) is a standard management protocol that allows network components (or any device or software with an SNMP agent) to report information concerning their status and activities. When this setting is enabled, the analysis server functions as an SNMP agent.

Note: Configure the SNMP Recipient! This setting has no effect unless an SNMP manager that will receive the traps has been configured. A message shows the status of the SNMP configuration for alerts. The status message is a link to the “SNMP Settings Screen“ where you can configure the settings.

The SNMP traps contain details of the threshold violation. Here is an example using Trap Receiver:

ServiceNow Settings

This area contains a single setting that enables sending an alert notification to the ServiceNow IT service management platform. This setting has no effect unless a ServiceNow user has been configured as a recipient in the “ServiceNow Integration Settings Screen“. A message shows the status of the ServiceNow configuration for alerts. The status message is a link to the screen where you configure the settings. Customize the content of the ServiceNow alert in the “Alert Notification Template Configuration Screen“.

Scheduling

These settings specify a blackout period that limits how violations trigger this alert.

Settings

Checkboxes control whether to trigger the alert at all or to only suppress email notification:

 Do not trigger this alert at all during the blackout period If this alert triggers (begins) during the blackout period, APM will not send an email notification or record it in the “Alert History Screen“

Aternity APM Version 11.4.2 109 Define an Alert Screen

 Trigger this alert but do not send email notification during the blackout period If this alert triggers during the blackout period, APM will not send an email but WILL record it in the Alert History screen. If an alert triggers before the blackout period and the alert clears during the blackout period, the alert-cleared email (if it is enabled in “Email Settings“) will still be sent. In both cases, once the blackout period is over, if the conditions for the alert are still met, the alert will be triggered then. The Blackout Period settings specify the times and days to the blackout is in effect. The From time starts on the days selected.

Example

Consider a blackout period that starts at 21:00 (9 p.m.) and ends at 23:00 (11 p.m.). For an alert that triggers after 5 minutes, the following cases illustrate how the blackout period works.

 Alert triggered before and partially overlaps blackout period Alert triggers at 8:30: Email sent Alert clears at 9:30: Email sent

 Alert entirely within blackout period Alert triggers at 9:30: Email not sent Alert clears at 10:00: Email not sent

 Alert triggered during and partially overlaps blackout period Alert triggers at 10:30: Email not sent At 11:00:00: Email sent (since the alert is still open and blackout is over) Alert clears at 11:30: Email sent

 Alert triggered during and partially overlaps blackout period Threshold violation begins at 10:54 Alert triggers at 10:59 (violation period started at 10:54): Email not sent At 11:00:00: Email sent (since the alert is still open and blackout is over) Alert clears at 11:30: Email sent

 Alert triggered during and partially overlaps blackout period Threshold violation begins at 10:58 Alert triggers at 11:03 (violation period started at 10:58): Email sent Alert clears at 11:30: Email sent

 Alert triggered before and completely overlaps blackout period Alert triggers at 8:30: Email sent At 9:00:00: Email not sent (entering blackout period does not clear the alert) Alert clears at 11:30: Email sent

110 Aternity APM Version 11.4.2 CHAPTER 13 Alert Notification Template Configuration Screen

Overview

As described in the “Define an Alert Screen“ topic, alert definitions can specify settings that trigger different notifications when threshold violations occur. (See the “Threshold Overview Screen“ topic for details on thresholds.) This screen specifies templates that control the content of the following types of alert notifications: – Email messages to addresses specified in the “Email Settings“ area of the “Define an Alert Screen“ – ServiceNow incident alerts, if enabled in the “ServiceNow Settings“ area of the “Define an Alert Screen“ Templates use the Jinja template engine and include variables that are replaced with alert-specific values. There are two templates, one used for when an alert is triggered and one for when it clears. (APM sends email for cleared alerts only if you enable Send Email once alert is cleared in the “Email Settings“ for the alert definition.) Typically, you do not need to adapt the default templates. However, you can adapt them to suit your purposes. You can add or change the HTML, use different variables, or use Jinja for conditional logic and simple math on values. Keep in mind that any changes you make to the templates will be reflected in all email and ServiceNow notifications. Click Validate button to validate the Jinja syntax in the template. To see how changes you make will appear in email, use the Send Test Email button in the “Email Settings“ screen for the alert definition. This sends email to recipients of the alert. Note that the test email uses hard-coded sample values for the variables specified in the template. Values for the alert name or description variables, for instance, will not reflect the actual alert.

Aternity APM Version 11.4.2 111 Alert Notification Template Configuration Screen

Template Variables

To refer to variables in the template, enclose them in double pairs of curly braces. For example, this line in the template: alert.name: {{alert.name}}

Generates the following output in the test email sent by clicking Send Test Email button in the “Email Settings“ for the alert definition. alert.name: CPU Usage

Not all variables are applicable to all alert notifications. For example, several variables are specific to notifications that an alert has cleared. In such cases, APM uses None as the variable value. The following table lists available variables with a brief description and the sample value sent in test email.

Variable Description Sample Value

analysis_server.hostname Analysis server host name https://my.analysis.server.riverbed.co m analysis_server.version Analysis server version 10.5.96 analysis_server.ip Analysis server IP address 172.16.1.1 alert.name Alert definition name CPU Usage alert.description Alert definition description Monitors the CPU usage and will alert if going outside of the set thresholds alert.email_settings E-mail recipients, message Emails will be sent as JSON/Human format settings Readable to the following recipients: [email protected] alert.dampening.minutes Transactional alerts 2 consecutive-minutes setting

alert.dampening.percent_ Transactional alerts ‘% violating’ 80 violation setting alert.dampening.minimu Transactional alerts ‘at least x 10 m_number transactions’ setting

alert.above_threshold True if entire violation period True above threshold

alert.severity Major if entire violation period minor was major, otherwise minor alert.current_value.severit Current value violating major or major y minor threshold

alert.current_value.above_ Current value above upper True threshold threshold

alert.url Contextual link back to Analysis https://my.analysis.server.riverbed.co Server m/#alert-history alert.current_value.above_ >= or <= comparing current >= or_below_threshold value to threshold

alert.current_value.thresh Threshold value being violated 50 old

112 Aternity APM Version 11.4.2 Alert Notification Template Configuration Screen

Variable Description Sample Value

alert.current_value Average value of the metric for 89 the last minute of the violation period

alert.average_value Average value of the metric over 30 the entire violation period

alert.unit Unit for violating metric % alert.source.type Type of alert source (e.g., component transaction type)

alert.source.name Name of alert source CTServer (beth01.tfnportal) disk2 alert.source.component Component of alert source disk2 alert.source.server Server portion of name beth01.tfnportal alert.source.instance Instance portion of name CTServer alert.source.server_tags Server tags of source application:Tradefast, deployment_group:Infrastructure, instance:Web Server alert.source Concatenates source type, name, component CTSecure (beth01.tfnportal) and component disk2

alert.violation.duration Duration since violation started 20 alert.trigger.time Time alert was issued 2016-07-25 18:06:41 alert.violation.start Time violations began 2016-07-25 17:46:41 alert.trigger.transactions Number of transactions None violating

alert.trigger.transactions_a Apdex score of transactions None pdex alert.trigger.number_mino Number of minutes considered 1 r minor violations

alert.trigger.number_majo Number of minutes considered 2 r major violations

alert.trigger.minutes Total minutes violating 3 alert.trigger.percent Percent of transactions violating None alert.trigger.current_value Metric value when alert was 89 issued alert.trigger.average_value Average value when alert was 56 issued

alert.blackout.enabled Whether blackouts are enabled False alert.blackout.from_secon Alert blackout start 0 ds

alert.blackout.to_seconds Alert blackout end 3600 alert.blackout.timezone Alert blackout timezone (valid UTC time zone values are case sensitive) alert.blackout.sunday Blackout applies to Sunday False

Aternity APM Version 11.4.2 113 Alert Notification Template Configuration Screen

Variable Description Sample Value

alert.blackout.monday Blackout applies to Monday True alert.blackout.tuesday Blackout applies to Tuesday False alert.blackout.wednesday Blackout applies to Wednesday True

alert.blackout.thursday Blackout applies to Thursday False alert.blackout.friday Blackout applies to Friday True alert.blackout.saturday Blackout applies to Saturday False threshold.upper Upper threshold • Minor: >50 percent; Major: >80 percent threshold.lower Lower threshold • Minor: <10 percent; Major: <5 percent threshold.upper.major Upper major threshold 80 threshold.upper.minor Upper minor threshold 50 threshold.lower.major Lower major threshold 5 threshold.lower.minor Lower minor threshold 10 threshold.metric.name Violating metric name CPU Usage threshold.metric.method Metric measurement method average (e.g., average)

threshold.type Threshold type: environmental environmental or transactional thresholds Values of all threshold in • Minor: <10 percent; Major: <5 definition percent • Minor: >50 percent; Major: >80 percent alert.clear.time Time violation period ended 2016-07-25 18:08:55 alert.clear.current_value Metric value after violation 30 period ended

alert.clear.average_value Average value after violation 28 period ended alert.total.time Final minute of alert 2016-07-25 18:08:45 alert.total.number_minor Final number of minutes 2 considered minor violations

alert.total.number_major Final number of minutes 3 considered major violations

alert.total.minutes 5 alert.total.percent Final percent of transactions None violating alert.total.transactions Final number of transactions None violating

alert.total.average_value Final average value 44 alert.total.transactions_ap Final Apdex score None dex

114 Aternity APM Version 11.4.2 Alert Notification Template Configuration Screen

Variable Description Sample Value

alert.total.current_value Final minute value 45 alert.duration Length of alert in seconds 74

Example: Show All Variables and Values

This example lists all of the available variables and their values. You can copy and append it to the default email templates to learn about the variables and how their values vary depending on the context of the alert (environmental or transactional, details of the thresholds, triggered or cleared, and so on). Keep in mind that any changes you make to the templates will be reflected in all notifications.

List of all variables and their values:

analysis_server.hostname: {{analysis_server.hostname}}
analysis_server.version: {{analysis_server.version}}
analysis_server.ip: {{analysis_server.ip}}
alert.name: {{alert.name}}
alert.description: {{alert.description}}
alert.email_settings: {{alert.email_settings}}
alert.dampening.minutes: {{alert.dampening.minutes}}
alert.dampening.percent_violation: {{alert.dampening.percent_violation}}
alert.dampening.minimum_number: {{alert.dampening.minimum_number}}
alert.above_threshold: {{alert.above_threshold}}
alert.severity: {{alert.severity}}
alert.current_value.severity: {{alert.current_value.severity}}
alert.current_value.above_threshold: {{alert.current_value.above_threshold}}
alert.url: {{alert.url}}
alert.current_value.above_or_below_threshold: {{alert.current_value.above_or_below_threshold}}
alert.current_value.threshold: {{alert.current_value.threshold}}
alert.current_value: {{alert.current_value}}
alert.average_value: {{alert.average_value}}
alert.unit: {{alert.unit}}
alert.source.type: {{alert.source.type}}
alert.source.name: {{alert.source.name}}
alert.source.component: {{alert.source.component}}
alert.source.server: {{alert.source.server}}
alert.source.instance: {{alert.source.instance}}
alert.source.server_tags: {{alert.source.server_tags}}
alert.source.transaction_type: {{alert.source.transaction_type}}
alert.source: {{alert.source}}
alert.violation.duration: {{alert.violation.duration}}
alert.trigger.time: {{alert.trigger.time}}
alert.violation.start: {{alert.violation.start}}
alert.trigger.transactions: {{alert.trigger.transactions}}
alert.trigger.transactions_apdex: {{alert.trigger.transactions_apdex}}
alert.trigger.number_minor: {{alert.trigger.number_minor}}
alert.trigger.number_major: {{alert.trigger.number_major}}
alert.trigger.minutes: {{alert.trigger.minutes}}
alert.trigger.percent: {{alert.trigger.percent}}
alert.trigger.current_value: {{alert.trigger.current_value}}
alert.trigger.average_value: {{alert.trigger.average_value}}
alert.blackout.enabled: {{alert.blackout.enabled}}
alert.blackout.from_seconds: {{alert.blackout.from_seconds}}
alert.blackout.to_seconds: {{alert.blackout.to_seconds}}

Aternity APM Version 11.4.2 115 Alert Notification Template Configuration Screen

alert.blackout.timezone: {{alert.blackout.timezone}}
alert.blackout.sunday: {{alert.blackout.sunday}}
alert.blackout.monday: {{alert.blackout.monday}}
alert.blackout.tuesday: {{alert.blackout.tuesday}}
alert.blackout.wednesday: {{alert.blackout.wednesday}}
alert.blackout.thursday: {{alert.blackout.thursday}}
alert.blackout.friday: {{alert.blackout.friday}}
alert.blackout.saturday: {{alert.blackout.saturday}}
threshold.upper: {{threshold.upper}}
threshold.lower: {{threshold.lower}}
threshold.upper.major: {{threshold.upper.major}}
threshold.upper.minor: {{threshold.upper.minor}}
threshold.lower.major: {{threshold.lower.major}}
threshold.lower.minor: {{threshold.lower.minor}}
threshold.metric.name: {{threshold.metric.name}}
threshold.metric.method: {{threshold.metric.method}}
threshold.type: {{threshold.type}}
thresholds: {{thresholds}}
alert.clear.time: {{alert.clear.time}}
alert.clear.current_value: {{alert.clear.current_value}}
alert.clear.average_value: {{alert.clear.average_value}}
alert.total.time: {{alert.total.time}}
alert.total.number_minor: {{alert.total.number_minor}}
alert.total.number_major: {{alert.total.number_major}}
alert.total.minutes: {{alert.total.minutes}}
alert.total.percent: {{alert.total.percent}}
alert.total.transactions: {{alert.total.transactions}}
alert.total.average_value: {{alert.total.average_value}}
alert.total.transactions_apdex: {{alert.total.transactions_apdex}}
alert.total.current_value: {{alert.total.current_value}}
alert.duration: {{alert.duration}}

116 Aternity APM Version 11.4.2 CHAPTER 14 SNMP Settings Screen

Overview

As described in the “Define an Alert Screen“ topic, the “SNMP Settings“ area enables sending an SNMP trap when threshold violations trigger an alert. The Simple Network Management Protocol (SNMP) is a standard management protocol that allows network components (or any device or software with an SNMP agent) to report information concerning their status and activities. When this setting is enabled, the analysis server functions as an SNMP agent. See the following link for an introduction to SNMP: https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol#Protocol_details Settings in this screen specify the SNMP manager that will receive these SNMP traps. Use the “Send Test“ button to verify that the settings are configured correctly. To interpret the traps, the SNMP manager that receives the traps must load the Management Information Base (MIB) that describes the structure of the alert data. Click the “Download alert SNMP MIB“ link to download the alert MIB.

SNMP Recipient

These settings specify details about the SNMP manager you want to receive SNMP traps when alerts are triggered. Besides identifying the SNMP manager (in the “Destination Address“ and “Destination Port“ settings), additional settings specify details of the trap notifications. You need to be familiar with how the manager is configured so you can to supply valid values for these settings.

Destination Address

The IP address or name of the system on which the SNMP manager is running.

Aternity APM Version 11.4.2 117 SNMP Settings Screen

Destination Port

The port on which the SNMP manager is listening for traps. The default is 162, the well-known port for SNMP.

SNMP Version

The version of SNMP to use:

 v2c to use Community-Based SNMP Version 2, or SNMPv2c (external link to more detail).

 v3 to use SNMP Version 3, or SNMPv3 (external link to more detail).

Security Level (SNMPv3 only)

This setting appears if you choose v3 as the “SNMP Version“. It specifies the user-based security model (USM) to use:

 noAuthNoPriv: Communication without authentication and privacy

 authNoPriv: Communication with authentication and without privacy

 authPriv: Communication with authentication and privacy The USM is described in detail by RFC 2574. See the following links for an introduction: http://www.net-snmp.org/tutorial/tutorial-5/commands/snmpv3.html https://www.webnms.com/simulator/help/sim_network/netsim_conf_snmpv3.html Each USM is successively safer and has additional settings to configure. The following table summarizes the additional settings required for each security level.

Security Level Required Fields Notes

noAuthNoPriv Username Required even without authentication.

Context Name In combination with an SNMP engine ID (the “Security Engine ID“ setting) , identifies a collection of information accessible by an SNMP entity. See RFC 5343 for more detail. AuthNoPriv Same as for “noAuthNoPriv“, plus:

Authentication Protocol Choose the protocol to use.

Authentication Password Authentication password for the user specified by “Username“. AuthPriv Same as for “AuthNoPriv“, plus:

Privacy Protocol Choose the protocol to use.

Privacy Password Privacy password for the user specified by “Username“.

118 Aternity APM Version 11.4.2 SNMP Settings Screen

Community String (SNMPv2c only)

This setting appears if you choose v2c as the “SNMP Version“. The community string controls access from an SNMP manager to an agent. Typically you do not need to supply a value here for the analysis server to send alert traps to an SNMP manager.

Security Engine ID

The SNMP engine ID is a unique string used to identify an agent or manager for administration purposes. SNMPv3 uses engine IDs in combination with a “Context Name“ to automatically discover and potentially limit communication to known SNMP entities. See RFC 5343 for more detail. If you supply a value here, it must be between 10 and 64 hexadecimal characters.

Configuration Status

This shows the status of the last “Send Test“ operation, if any. Possible values are:

 Unset: The “SNMP Recipient“ values have not been set.

 Untested: The SNMP Recipient values have been set but the Send Test operation has not been attempted.

 Tested (no acknowledgement): The Send Test operation did not receive an inform request acknowledgment from the SNMP manager. This does not necessarily mean the SNMP Recipient settings are incorrect (see “Send Test“ for details).

 Tested successfully: The Send Test operation successfully received an inform request acknowledgment from the SNMP manager, indicating the SNMP Recipient settings are correct.

Download alert SNMP MIB

The SNMP manager specified in the “SNMP Recipient“ area must load the Management Information Base (MIB) that describes the structure of the alert data. Click the link to download the alert MIB. The file is named appinternals_alert_snmp_mib__.mib and is saved in your browser’s download area (typically accessible via the CTRL+J keyboard shortcut).

Send Test

Click Send Test to verify the settings in the “SNMP Recipient“ area. The analysis server sends up to three SNMP “inform request” notifications to the SNMP manager specified in the settings. For the test to succeed, the SNMP manager must acknowledge the inform request. If the analysis server does not receive the inform request acknowledgment from the SNMP manager, it sets the “Configuration Status“ to Tested (no acknowledgement). This does not necessarily mean that SNMP Recipient settings are incorrect. It is possible that the acknowledgment from the SNMP manager to the analysis server was somehow blocked (by a firewall, for example) or that the manager was not configured to send acknowledgments.

Aternity APM Version 11.4.2 119 SNMP Settings Screen

If you have access to the SNMP manager and you can see the inform requests, that means the SNMP Recipient settings are correct. The following example shows an inform request from the Send Test operation in Trap Receiver:

Save / Revert

Save and Revert are enabled after you change any settings:

 Click Save to save settings.

 Click Revert to return to the previously-saved settings.

120 Aternity APM Version 11.4.2 CHAPTER 15 ServiceNow Integration Settings Screen

Overview

As described in the “Define an Alert Screen“ topic, you can specify settings that trigger an alert notification when threshold violations occur. (See “Threshold Overview Screen“ for details on thresholds.) Such notifications can be sent to different destinations, including to the ServiceNow IT service management platform. The content and formatting of the alert notification is controlled by the “Alert Notification Template Configuration Screen“. For ServiceNow notifications to work, you must have access to a ServiceNow instance that has the Riverbed SteelCentral AppInternals application installed. Download the application from the ServiceNow store at https://store.servicenow.com. Settings in this screen configure the ServiceNow instance and account that APM uses to send notifications.

Aternity APM Version 11.4.2 121 ServiceNow Integration Settings Screen

Notifications appear as incident alerts in the ServiceNow application:

ServiceNow Alert Recipient

Destination Address

URL address for the ServiceNow instance to which to send the notifications. Each instance has a unique address (usually http://.service-now.com)

Destination Port

The network port for the ServiceNow instance. 443 is the default.

Username

The user name to use to log in to the ServiceNow instance. ServiceNow recommends creating a new user for this purpose and to limit its privileges by selecting the Web service access only and Internal Integration User options.

122 Aternity APM Version 11.4.2 ServiceNow Integration Settings Screen

After creating the user, add the special role x_rivt2_aix_alerts.user that allows it to receive APM alert notification data. Click the Roles tab and Edit to open the Edit Members screen and add the role. This role does not appear in the list of roles in the ServiceNow Edit screen. Start typing the role name in the Collection field to add it:

Password

The password for the ServiceNow user specified in “Username“.

Send Test

Click Send Test to send a test alert notification to ServiceNow. If the test succeeds, the Configuration Status changes from Untested to Tested Successfully. In addition, the ServiceNow application should show a TEST ALERT record similar to the following:

Save / Revert

Save and Revert are enabled after you change any settings:

 Click Save to save settings.

Aternity APM Version 11.4.2 123 ServiceNow Integration Settings Screen

 Click Revert to return to the previously-saved settings.

124 Aternity APM Version 11.4.2 CHAPTER 16 Threshold Overview Screen

Overview

This screen has tabs for:

 Environmental and Transactional “Thresholds“ that specify acceptable limits for various metric values

 “Default Coloring“ settings that control the default coloring of some cards

Thresholds

Thresholds specify acceptable limits for different metrics that APM monitors. As described in “Threshold Limits“, thresholds can specify up to four limits. If a metric value falls outside of these limits, APM generates a corresponding threshold violation. APM evaluates thresholds based on the average of 1-second metric values over the last minute. Threshold violations appear in various tables and graphs in the user interface. In addition, threshold violations can potentially trigger alerts. Alerts cause APM to record the alert in the “Alert History Screen“ and, if configured, send an email notification and SNMP trap with details about the alert. You define the conditions that control when threshold violations trigger an alert in the “Define an Alert Screen“, along with recipients for email notifications and SNMP traps. There are two types of thresholds, environmental and transactional:

 “Environmental Thresholds“ specify acceptable values for key operating system resource metrics or application instance memory usage. Each threshold specifies a metric and the name of a server or application instance it will be evaluated against (or [default]--see “[default] Thresholds“). Click the Environmental tab to see a list of any environmental thresholds that have been defined.

Aternity APM Version 11.4.2 125 Threshold Overview Screen

 “Transactional Thresholds“ specify acceptable metric values for transactions that match specific transaction types. Each threshold specifies a metric and the name of a transaction type it will be evaluated against (or [default]--see “[default] Thresholds“). (You define transaction types on a separate screen. See “Transaction Types“ for details.) Click the Transactional tab to see a list of any transactional thresholds that have been defined.

Threshold Limits

Each metric can specify up to four limits:

 A set of upper and lower limits that defines a major violation. If the average metric value over a minute falls outside of these limits, APM generates a major threshold violation.

 A set of upper and lower limits that defines a minor violation. If the average metric value over a minute falls outside of these limits, APM generates a minor threshold violation. You can specify any combination of upper, lower, major, and minor limits. However, the values must increase relative to each other: Lower major < Lower Minor < Upper Minor < Upper Major When you define a threshold, clear the check box for any limits you do not want evaluated. For example, for response time thresholds, you typically will clear the lower limit check boxes, since short response times are not a concern:

[default] Thresholds

The [default] thresholds are special catchall thresholds. They apply only if there are no other thresholds defined for a particular server, application instance, or transaction type. You can create [default] environmental and transactional thresholds for each metric allowed for the threshold type. New installations create a [default] environmental threshold by default. This threshold will generate threshold violations on any server when CPU usage exceeds 80% (minor) and 95% (major). You can create additional [default] thresholds for different metrics, or change (or delete) the single [default] environmental threshold.

126 Aternity APM Version 11.4.2 Threshold Overview Screen

Adding, Modifying, or Deleting Thresholds

In either tab, to add or modify thresholds:

 To add a new threshold, click the add button above the thresholds list. The Add Threshold dialog opens.

 To edit an existing threshold, select the row for the threshold and click the edit icon ( ) in the row for the threshold you want to edit. The Edit Threshold dialog opens with the settings for that threshold.

 To delete an existing threshold, click the delete icon ( ) in the row for the threshold you want to delete.

Default Coloring

Click the “Default Coloring“ tab to change the limits that control coloring of the “Application Map Card“ and “Top Transaction Types Card“.

Environmental Thresholds

Environmental thresholds specify acceptable values for key operating system resource metrics or application instance memory usage. Each threshold specifies the name of a server or application instance it will be evaluated against. Click the Environmental tab to see a list of any environmental thresholds that have been defined. In a new installation, there is a single [default] environmental threshold (see “[default] Thresholds“). It specifies upper limits for acceptable CPU usage for servers. You can change its settings, define additional [default] thresholds for other metrics, or delete it altogether. Supply values as follows:

Server/Instance Choose whether this threshold will apply to a server or an application instance.

Server The name of a server or application instance. For that server or application instance, the Instance metric specified in the Metric setting will be evaluated against the specified limits. This name must match a server or application instance already known to the analysis server (as shown in the “Agent List Screen“ or “Process List Screen“), or use the special [default] name (see “[default] Thresholds“). The asterisk wildcard character (*) is also supported. For more information, see “Environmental Thresholds, The Wildcard Character, And Tags“ Tag An already defined key=value tag, as in container type=Docker. The asterisk wildcard character (*) is also supported. For more information, see “Environmental Thresholds, The Wildcard Character, And Tags“ Metric The metric on which to base the threshold.

Lower Major • The limits for acceptable values for this metric. Each metric can specify up to four Lower Minor limits, as described in “Threshold Limits“. Upper Minor Upper Major

Aternity APM Version 11.4.2 127 Threshold Overview Screen

Transactional Thresholds

Transactional thresholds specify acceptable metric values for transactions that match specific transaction types. (You define transaction types on a separate screen. See “Transaction Types“ for details.) Click the Transactional tab to show any user-defined transactional thresholds. (There are no default transactional thresholds.)

Metric Must be Compatible with Transaction Type Applies To Setting

To create a transactional threshold, you specify the name of an existing transaction type and a response time metric. Keep in mind that the metric and transaction type’s scope (its “Applies To“ setting) must be compatible:

 If the transaction type’s Applies To setting is Transaction Segments, do not choose Page Load Time as the Metric value. Page load data is not available for transaction segments and thresholds based on such transaction types will never trigger a violation.

 If the transaction type’s Applies To setting is Transaction Segments, do not choose End-to-End Response Time as the Metric value. This metric will never trigger a violation for transaction segments. Use the Server Response Time metric instead.

 If the transaction type’s Applies To setting is End-to-End Transactions, choosing Server Response Time as the Metric value will never trigger a violation if the application ONLY has page load data (in other words, it does not have transaction trace data).

Specifying Transactional Thresholds

There are no default transactional thresholds. For each transaction type, you specify the metric and acceptable values. Supply values as follows:

Transaction Type The name of a transaction type. For transactions that match the transaction type, the metric specified in the Metric setting will be evaluated against the specified limits. This name must match an existing transaction type (as shown in the “Transaction Types Screen“, or use the special [default] name (see “[default] Thresholds“). Metric The metric on which to base the threshold.

Lower Major The limits for acceptable values for this metric. Each metric can specify up to four limits, as Lower Minor described in “Threshold Limits“. Upper Minor Upper Major

128 Aternity APM Version 11.4.2 Threshold Overview Screen

Consider the following example:

APM evaluates any transactions that match the Home transaction type to see if their response time was .5 or 1 second or greater:

 If a transaction’s response time is more than .5 seconds but less than 1 second, the transaction type generates a minor threshold violation.

 If a transaction’s response time is 1second or greater, the transaction type generates a major threshold violation. APM keeps track of the number of transactional threshold violations for the transaction type and uses them to calculate Apdex (see “Transactional Thresholds and Apdex Scores“) scores.

Transactional Thresholds and Apdex Scores

APM also uses “Transactional Thresholds“ violations to calculate an Apdex score. Apdex scores are a standard method for reporting user satisfaction with software application performance. Scores range from 0 to 1, where 1 is the best possible score and zero is the worst. See the following link for background on Apdex scores: http://www.apdex.org/

How APM Calculates Apdex Scores

APM calculates Apdex scores every minute based on the count of transactions that match transaction types, and which of those transaction had minor and major violations. Each transaction is scored as follows:

 No violation: 1 point

 Minor violation: 1/2 point

 Major violation: 0 points

Aternity APM Version 11.4.2 129 Threshold Overview Screen

The Apdex score is the total points divided by the number of transactions for the minute: Apdex = (1.0 * no_violation_count + .5 * minor_violation_count)/number_transactions

Note: Without Transactional Thresholds, Apdex score Is Always 1 If there are no transactional thresholds defined, there can be no violations, which results in an Apdex score of 1.

How APM Uses Apdex Scores

APM uses Apdex scores as follows:

 In the “Apdex Card“ shown in some “Data Analysis Tabs and Metric Data Views“

 As a column in the “Alert History Screen“

 To color-code maps in the “Application Map Card“ and “Application Map Tab“. – If the Apdex value 0.85 or greater, it is green (good to excellent). – If the Apdex value 0.5 or greater, it is yellow (poor to fair). – If the Apdex value below 0.5, it is red (unacceptable).

Default Coloring

Settings here specify the default behavior for how metric values control colors in the following cards:

 “Geography Card“: the color of countries or regions, unless the display has been filtered by a transaction type.

 “Top Transaction Types Card“ and “Top Segment Transaction Types Card“: the color of transaction bubbles, if there are no “Transactional Thresholds“ defined for the transaction type. The Minor and Major settings affect colors as follows:

 Green—The average metric value is less than the Minor setting over the time range.

 Yellow— average value is equal or greater than the Minor setting but less than the Major setting.

 Red—The average metric value is equal or greater than the Major setting.

130 Aternity APM Version 11.4.2 CHAPTER 17 Applications Screen

This screen opens when you click the Application List option of the Configure menu. It lists any applications that have been defined and options to create and modify application definitions. Application definitions specify transaction types, servers, and instances whose data you want to group together. Once created, applications appear in the “Applications Tab“, where users can filter results by applications and see values of metrics across all applications and for individual applications.

Operations and Controls

If no applications have been defined, the Manage Applications table is empty. The table has the following controls:

 To add a new application definition, click Define an Application. The “Define an Application Screen“ opens.

 To edit an existing application definition, click the edit icon ( ) in the row for the application definition you want to edit. The “Define an Application Screen“ opens with the settings for that application definition.

 To delete an existing application definition, click the delete icon ( ) in the row for the application definition you want to delete. To delete many applications at once, follow the steps described in “Example: Deleting All Applications with Import“.

Aternity APM Version 11.4.2 131 Applications Screen

Bulk Editing Application Definitions with Export and Import

You can export all available application definitions and modify them in bulk using a spreadsheet or text editor. If you have many application definitions, this is easier than modifying one at a time.

Overview

To export transactions to a text file, click the Export button ( ) at the bottom of the list of application definitions. This creates and downloads a file with the application definitions exposed as blocks of comma-separated-value (CSV) records. The file is named appinternals_applications__.csv and is saved in your browser’s download area (typically accessible via the CTRL+J keyboard shortcut). After you modify the application definitions to suit your purposes, click the Import button ( ) at the bottom of the list of application definitions and specify the modified CSV file.

Note: Importing Deletes Non-Matching Application Definitions Import operations delete any existing application definitions on the analysis server that do not match names in the imported CSV file. Import operations update application definitions with matching names and create new application definitions for names in the CSV file that do not match existing application definitions on the analysis server.

This feature is similar to exporting and importing transaction types. See “Bulk Editing Transaction Types with Export and Import“ for usage notes and examples that also apply to exporting and importing application definitions.

132 Aternity APM Version 11.4.2 Applications Screen

Format

Each application definition is separated by a row of hash characters (#). Field names in the file are the same or similar to the corresponding field in the “Define an Application Screen“. The following example shows the correspondence between the fields in the Define an Application screen and an exported CSV file:

Example: Deleting All Applications with Import

You can import a file with a single application definition to delete all other application definitions. This is useful if you want to delete all application definitions in your environment. (Otherwise you must click the delete icon ( ) for each application definition.) Follow steps similar to these:

1) Optional: back up existing application definitions by clicking the Export button ( ) at the bottom of the list of transactions.

2) Create a CSV file similar to the following. It defines a single dummy application definition you import and then delete: "version","v1.0"

"###################" "name","Delete me" "description","Delete me to get rid of the last application" "topMatch","any" "criteria","transactiontype" "match","any" "pattern","na"

3) Click the Import button ( ) at the bottom of the list of transactions and specify the CSV file you created.

4) Click the delete icon ( ) for the single remaining Delete me application definition.

Aternity APM Version 11.4.2 133 Applications Screen

134 Aternity APM Version 11.4.2 CHAPTER 18 Define an Application Screen

Use this screen to define an application. Navigate here by clicking the Define an Application button in the “Applications Screen“. Application definitions specify transaction types, servers, and instances whose data you want to group together as a single application. Once created, applications appear in the “Applications Tab“ and users can specify them as “Global Filters“ that limit all displays to a particular application. An application definition is a series of criteria that matches the set of transaction types, servers, and instances you want, and associates them with a friendly application “Name“. Each criterion specifies at least one field and a corresponding pattern (for example, Servers and nh-trade* ). By combining multiple criteria and, within a criterion, multiple patterns, you can create flexible sets of patterns that match exactly the components you want to group together. Depending on the “Match any / Match all of the following criteria“ setting, data must match all criteria (Match all) or any one of them (Match any) to be included as part of the application.

Name

Required. This name appears in the “Applications Tab“ and in the “Applications Screen“. You cannot change this name after you first save the application definition.

Description

Optional. This name appears in the “Applications Screen“.

Aternity APM Version 11.4.2 135 Define an Application Screen

Add Hosts and Relationships

These options supplement the application map that APM automatically discovers and displays in the “Application Map Tab“. These options appear only when you modify an existing application definition. They are not available when you first define an application.

 “Add Host“ creates additional server nodes that otherwise would not appear in the map.

 “Add Relationship“ creates relationship arrows between servers and instances that otherwise would not show as connected in the map. These options are accessible directly from the application map if there is an application filter. See “Edit Mode (Must Filter by Application Definition and Choose ‘Physical’)“

Add Host

Use this option to add “Server Nodes“ that are not in the application map. If the server is already part of the automatically-discovered application map, adding it manually has no effect.

Add Host Click to add a server.

Server In the drop-down list, select the server to add.

Click to delete the server from the list.

Add Relationship

Use this option to add “Relationship Arrows“ between application instances that are not connected in the map. If the relationship is already part of the automatically-discovered application map, adding it manually has no effect.

Add Relationship Click to a relationship. APM adds a dotted-line relationship arrow between the two instances.

From In the drop-down list, select the server and AppDomain or JVM to connect from. The name in the list starts with the server name followed by a colon, then lists the web site name followed by #, then gives the AppDomain or JVM name. To In the drop-down list, select the AppDomain or JVM to connect to.

Click to delete the relationship in that row from the list.

136 Aternity APM Version 11.4.2 Define an Application Screen

To add a relationship arrow, follow these steps:

1. Click to expand the Add Relationships area.

➘ You then see an application map reflecting the most recent month’s worth of application activity.

2. When you see the Add Relationships application map, look below the map and click the Add Relationship button to add a relationship.

➘ A row with a From drop-down list to the left and a To drop-down list to the right appears.

3. In the From drop-down list, select the server and instance to connect from. The name you see is in this format: :#

4. On the same line, in the To drop-down list, select the server and AppDomain or JVM to connect to.

➘ Dotted lines draw the connection in the actual application map.

➘ Note that no remote application tiers are shown in the list, only servers and instances.

5. To add more relationships, repeat the preceding steps starting with step 2.

6. To delete a relationship, click the to the far right in that row.

7. Click Save to save the revised application definition.

8. When you next view the map in the “Application Map Tab“, the relationships you added manually will show in the map.

Aternity APM Version 11.4.2 137 Define an Application Screen

Match any / Match all of the following criteria

This area specifies the patterns that you want to match with the application name. Click “Add Criteria“ to specify one or more criteria that each specify patterns. Each criterion specifies a component (Instances, Transaction Types, or Servers) and one or more corresponding patterns to match it. By combining multiple criteria and, within a criterion, multiple patterns, you can create flexible sets of pattern matches using logical AND and OR operations:

 Between criteria, use the Match any / Match all of the following criteria setting to choose how APM determines whether data matches: – Match any is the default and means that if data matches criteria specified by any of the sections (“Instances“, “Transaction Types“, and “Servers“), it will be included as part of the application. APM performs a logical OR to determine a match. – Match all means that data must match criteria specified by all three sections to be included as part of the application. APM performs a logical AND to determine a match. This setting affects matching between criteria only.

 Within a criterion, use the “Match any / Match all of these “ setting to choose whether the specified component value matches any of the corresponding patterns (a logical OR operation) or must match all of them (a logical AND operation).

Add Criteria

When you add a new transaction, the screen opens with a single “Criterion Details“ area. Click Add Criteria to add more criteria.

Criterion Details

For each criterion, you must specify a component (Instances, Transaction Types, or Servers) and one or more patterns you want its value to match. If you specify multiple patterns, you also need to specify whether the field value matches any or all of them.

Match any / Match all of these

This setting has an effect only if you specify multiple patterns in a criterion:

 Choose any if the component value needs to match only one of the patterns (a logical OR operation).

 Choose all if the component value must match all of the patterns (a logical AND operation).

138 Aternity APM Version 11.4.2 Define an Application Screen

Instances, Transaction Types, or Servers

Choose a component. Patterns you specify for this criterion will be compared to values of the component to determine if the corresponding data will be included as part of the application.

Component Matches

Instances The name of a .NET app domain or Java JVM process that APM is monitoring. APM discovers processes automatically and lists them in the “Agent Details Screen“ where you can choose to “instrument” and monitor them. Instance names are the also matched by the “instance“ search field in the Search tab. Transaction Types The name of a transaction type shown in the “Transaction Types Screen“. Transaction type names are also matched by the “transactiontype“ search field in the Search tab. Servers The system name of a system running the APM agent software that generated the trace data. Sever names are also matched by the “server“ search field in the Search tab.

Pattern to Match

Specify the pattern you want the value of the component to match. To specify multiple patterns that the component value must match, click the + button.

 Enter a space to see values for existing instances, transaction types, or servers. You can accept any value and modify it with wildcards to suit your purposes. For example:

 Patterns are not case sensitive.

Aternity APM Version 11.4.2 139 Define an Application Screen

 Patterns can use an asterisk (*) as a wildcard to match zero or more characters and a question mark (?) to match a single character. Use a backslash (\) as an escape character before either wildcard to match that character in a string. You can use wildcards anywhere within the pattern. For example, the following pattern will match systems named vh6-rh5-4 and vh8-w2k8-1, but not vh14-w2k8-3 or ttw1: vh?-*-?

Save

Click Save to save the application definition and return to the “Applications Screen“.

140 Aternity APM Version 11.4.2 CHAPTER 19 Transaction Types

Overview

A transaction type is a series of criteria that matches some set of interesting transactions and associates them with a friendly transaction type “Name“. Each criterion specifies at least one field and a corresponding pattern (for example, URLs and http://*/tradefast/*.aspx* ). By combining multiple criteria and, within a criterion, multiple patterns, you can create flexible sets of patterns that match exactly the transactions you want to group together. APM supplies a set of “Default Transaction Types“ that serve as examples and provide catch-all criteria to make sure that all transactions will match at least one transaction type.

How APM Exposes Transaction Types

APM shows transaction types directly in “Data Analysis Tabs and Metric Data Views“ and also uses them indirectly to configure application definitions, alerts, and thresholds. Transaction types are used as follows:

 “Overview Tab“: The “Top Transaction Types Card“ shows a bubble chart with the transaction types that matched the most transactions.

 “Transaction Types Tab“: Displays a table and related cards with details about transactions that match transaction types.

 “Search Tab“: – Users can specify the name as the “transactiontype“ search field value. – The name of transaction types appear in the Transaction Type column of the results table of the “Search Tab“.

 “Define an Application Screen“: Application definitions can specify transaction type name as one of the criteria that define an application.

 “Transactional Thresholds“ specify a transaction type, metric, and threshold values. If a matching transaction violates a threshold (in other words, the metric value falls outside the specified thresholds), APM: – Evaluates conditions specified in the “Define an Alert Screen“ for a possible alert notification. – Reports the violation in the “Violations Card“ on various data analysis tabs.

Aternity APM Version 11.4.2 141 Transaction Types

APM also uses transactional threshold violations to calculate Apdex scores, as described in “Transactional Thresholds and Apdex Scores“.

Default Transaction Types

APM supplies a set of default transaction types. These defaults:

 Insure that display elements that rely on the presence of transaction types (see “How APM Exposes Transaction Types“) will be populated with data. The “Catch-all“ definitions in the default transaction types guarantee that elements such as the “Top Transaction Types Card“ will never be empty:

 Provide a set of examples to learn about transaction types. Use the defaults as models for your own transaction types or modify them to suit your environment. The Aternity icon ( ) next to a name in the “Transaction Types Screen“ and “Define a Transaction Type Screen“ indicates that the transaction type is a default transaction type.

Default Transaction Type Blocks

The default transaction types use “Transaction Type Precedence“ values to create blocks of transaction types for different purposes.

Precedence Block Purpose

1-999 Pre-Filtration These transaction types are evaluated first and specify URL patterns that match uninteresting file extensions (such as .js, .css, .pdf, .doc). Add transaction types patterns to this block to filter out files that would otherwise clutter user transaction types with unwanted matches. 1000-7999 User-defined Use this precedence block for transaction types that match some set of interesting transactions specific to your environment. When you create transaction types in the “Define a Transaction Type Screen“, APM assigns them a precedence value in this block.

142 Aternity APM Version 11.4.2 Transaction Types

Precedence Block Purpose

8000-9997 Path-based These transaction types specify URL patterns that are typically interesting in most environments. Their higher precedence means they are evaluated after user transaction types. 9998-9999 Catch-all These transaction types specify very broad patterns that will match all transactions that do not match any of the other blocks. They guarantee that every transaction will match at least one transaction type.

Using these precedence conventions for grouping transaction types is a useful way to organize transaction types. However, you can change or ignore the conventions, or delete the default transaction types altogether, to suit your purposes. The following figure shows the precedence blocks for default transaction types in the Manage Transaction Types screen:

Aternity APM Version 11.4.2 143 Transaction Types

Adding Default Transaction Types

New installations of the analysis server automatically add the default transaction types. The behavior for upgrades from versions earlier than Version 10.12.0 depends on whether there are existing transaction types:

 If there are no existing transaction types, the upgrade automatically adds the default transaction types.

 If there are existing transaction types, the upgrade increases their “Transaction Type Precedence“ by 1000 but does not automatically add the default transaction types. Instead, the Add Defaults button ( ) appears at the bottom of the list of transactions in the “Transaction Types Screen“. Click Add Defaults to add the default transaction types. The Add Defaults button disappears after you successfully add default transaction types. The Add Defaults operation fails if any of the existing transaction types have the same name or precedence. For example:

The Download default Transaction Types as a CSV file (requires internet access) link downloads the latest default transaction types from a Aternity server. If the Add Defaults operation fails, you can download the default transaction types by clicking the link. Edit the downloaded file to change the conflicting transaction type names and then import it. See “Bulk Editing Transaction Types with Export and Import“ for details.

What are “End-to-End Transactions” and “Transaction Segments”?

In addition to specifying patterns that match interesting transactions, all transaction types must specify the scope of the performance data they are evaluated against. The scope of the data indicates which part of a potentially-complex series of related transactions you are interested in investigating. The scope of the data is related to whether APM generates it from monitoring applications or web pages:

 Application performance: APM monitors application performance at a JVM (for Java) or appdomain (for .NET) level. JIDA or the dotNet sub-agent instrument processes on the application tiers where they are configured. (You enable instrumentation by choosing “processes to instrument” in the “Agent Details Screen“ screen.) This application monitoring generates transaction trace data at the JVM or appdomain level. APM detects related transactions from different traces on multiple application tiers and “stitches” them together.

 Web page performance: APM monitors web page performance by collecting data on page loads and (if configured in the “Customize Snippet“ options of the “Define a Configuration Screen“) AJAX requests in users’ web browsers. This data reflects application performance from the perspective of the end user.

144 Aternity APM Version 11.4.2 Transaction Types

The options available in the “Applies To“ setting specify which type of data the transaction type will be evaluated against:

 Transaction Segments: Evaluate against application performance data only, and only the individual, “unstitched” transaction traces. This data is always specific to a single server and will never include web page performance data.

 End-to-End Transactions: Evaluate against web page performance data and stitched application performance data. Do not evaluate against unstitched transaction traces.

 End-to-End Transactions and Transaction Segments: Evaluate against web page performance data, stitched application performance data, and unstitched tra.nsaction traces.

“Transaction Segments” Not Allowed with Certain Criteria

Note that some fields in the “Match Field List“ are not allowed with Transaction Segments and End-to-End Transactions and Transaction Segments. Specifying these fields would result in a transaction type that never matches any transaction segments. Although the transaction type could potentially match end-to-end trace data, to avoid confusion, APM does not allow you to specify these fields with either option that includes Transaction Segments:

 Any field that is generated from web page performance data (such as “Page Titles“). These fields will never match data in a transaction segment.

Aternity APM Version 11.4.2 145 Transaction Types

 The “Servers“ field, in a criterion that specifies Match all and more than one pattern to match. Transaction segment data always refers to a single server, so specifying that a transaction must match multiple server patterns will never match data in a transaction segment:

Creating Transaction Types

This section describes different approaches to creating transaction types:

 “Modifying Default Transaction Types“

 “Right-Clicking Values in Analysis Screens“

 “Creating a Simple Transaction Type from Scratch“

 “Bulk Editing Transaction Types with Export and Import“

Modifying Default Transaction Types

Use the “Default Transaction Types“ as examples. You can modify them to suit your environment or use them as models to create new transaction types.

Right-Clicking Values in Analysis Screens

Note: Administrative Accounts Only The right-click options described here are available for administrative accounts only.

146 Aternity APM Version 11.4.2 Transaction Types

A useful shortcut for creating transaction types is to right-click on values such as servers or URLs in the “Search Tab“ or in the “Transaction Details“ window. For example, right-clicking a URL in the “Search Results“ table:

1) Click the Create a Transaction Type option to open the “Define a Transaction Type Screen“.

2) The “Add Criteria“ area is filled in with the value of the selected URL. You can modify the criterion and add others to suit your purposes.

3) Supply a useful name (you cannot rename transaction types) and click Save. Values that support the Create a Transaction Type option by right-clicking include the following: In the “Search Tab“:

 “Search Results“ table: –URL –URL Path – Instance –Server – Page Title – Page Tag 1, 2, 3

 “Refine Results“ panel: –URL Path – Instance –Server In the “Transaction Details“ window:

Aternity APM Version 11.4.2 147 Transaction Types

 “Overview Tab“: –URL – Page Title –Page Tags

 “URLs Tab“: –URL –URL Outbounds –Server – Instance

 “Top Calls and Remote Calls Tab“: –Class –Method – Category

 “Call Tree Tab“: – Instances – Categories –Class –Method –Process Users –Servers –SQL Families – SQL Statements/Procedures –URLs –URL Outbounds –Web Service Methods

 “Exceptions Tab“: –Class –Method –Server – Instance

 “SQL Tab“: – SQL Statements/Procedures

 “AJAX Tab“: –URL

148 Aternity APM Version 11.4.2 Transaction Types

Creating a Simple Transaction Type from Scratch

This example shows how to create a simple transaction type. This example matches any invocation of the orders.aspx ASP.NET page:

1) In the “Transaction Types Screen“, click Add a Transaction Type.

2) The “Define a Transaction Type Screen“ opens. Supply a transaction type name (Orders)

3) In the “Add Criteria“ area choose url in the Match any of these list and supply a pattern of */orders.aspx:

4) Click Save.

Aternity APM Version 11.4.2 149 Transaction Types

In the “Search Tab“, users can then specify the Orders transaction type to search for that URL pattern:

Bulk Editing Transaction Types with Export and Import

You can export all available transaction types and modify them in bulk using a spreadsheet or text editor. If you have many transaction types, this is easier than modifying one at a time.

Overview

To export transactions to a text file, click the Export button ( ) at the bottom of the list of transactions in the “Transaction Types Screen“. This creates and downloads a file with the transaction types exposed as blocks of comma-separated-value (CSV) records. The file is named appinternals_transactiontypes__.csv and is saved in your browser’s download area (typically accessible via the CTRL+J keyboard shortcut). After you modify the transaction types to suit your purposes, click the Import button ( ) at the bottom of the list of transactions and specify the modified CSV file.

Note: Importing Deletes Non-Matching Transaction Types Import operations delete any existing transaction types on the analysis server that do not match names in the imported CSV file. Import operations update transaction types with matching names and create new transaction types for names in the CSV file that do not match existing transaction types on the analysis server.

150 Aternity APM Version 11.4.2 Transaction Types

Cases where exporting and importing transaction types is useful include:

 Modifying and Reimporting You can edit the CSV file to make changes to multiple transaction types at the same time. When you import the file (either on the same analysis server or on another), the transaction types will exactly reflect those in the file. Only transaction types with matching names are actually modified. Keep in mind the following implications of this behavior: – Changing names does not rename. You can change names of transaction types in the exported file. While this can be useful, it does not actually rename the transaction type. On import, the transaction type with the original name is deleted and a new transaction is created with the new name. This means that all existing transaction data that matches the transaction type will still be associated with the old name. Only new data will be associated with the new name. – Importing deletes all transaction types in the analysis server whose names do not match names in the CSV file. You can easily back up before importing changes by first exporting.

 Copying transaction types from one analysis server to another You can easily copy transaction types between analysis servers. Simply export from one and import to the other.

 Configuring large numbers of transaction types for the first time You can quickly create many transaction types from scratch by editing the CSV file. The “Example: Creating Transaction Types for the Top 10 URLs“ shows one approach.

 Deleting all transaction types quickly You can import a file with a single transaction type to delete all other transaction types. See “Example: Deleting All Transaction Types with Import“ for details.

Format

Each transaction type is separated by a row of hash characters (#). Field names in the file are the same or similar to the corresponding field in the “Define a Transaction Type Screen“. For example:

Aternity APM Version 11.4.2 151 Transaction Types

Comments in the exported CSV file describe the syntax and list valid values for fields. In summary:

 Fields and values are not case sensitive.

 Import operations ignore blank lines and lines beginning with # .

 The precedence field specifies the precedence of the transaction type (see “Transaction Type Precedence“).

 Supply one or more sets of "criteria" fields. Each "criteria" set must have the "match" field and one or more "pattern" fields.

 Valid values for the "appliesTo" field correspond to values in the “Applies To“ list in the user interface:

UI Applies To Values Corresponding "appliesTo" Values

End-to-End Transactions "appliesTo","endtoend_and_local" and Transaction Segments End-to-End Transactions "appliesTo","end_to_end_trace" Transaction Segments "appliesTo","local_code_trace"

 Valid values for the "match" field are "any" and "all". They correspond to values in the “Match any / Match all“ setting.

 Valid values for the "criteria" field correspond to values in the “Match Field List“ list. However, the names are different in the CSV (they are the same as search field names described in “Search Field Reference“):

UI Criteria Values Corresponding "criteria" Values

Categories "criteria","category" Class.method.parameters "criteria","class.method.parameter" Class.methods "criteria","class.method" Classes "criteria","class" HTTP Headers "criteria","header" Instances "criteria","instance" Page Tags "criteria","pagetags" Page Titles "criteria","pagetitle" Process Users "criteria","processuser" Servers "criteria","server" SQL Families "criteria","sqlfamily" SQL Statements/Procedures "criteria","sql"

URL Outbounds "criteria","urloutbound" URL Parameters "criteria","urlparameter" URLs "criteria","url" Web Service Methods "criteria","webservicemethod"

152 Aternity APM Version 11.4.2 Transaction Types

Example: Creating Transaction Types for the Top 10 URLs

Suppose you want to create transaction types for each of the 10 most-frequently observed URL patterns in your environment. Creating many similar transaction types in the “Define a Transaction Type Screen“ is tedious. Instead, follow steps similar to these:

1) Get the patterns you want to embed in transaction types with the “transactioncount“ analysis operator in the “Search Tab“. Enter the following search string: | transactioncount -group_by urlpath -limit 10 This search might return patterns such as the following: /tier1client/txn3/tier1_2/tier2_3/tier3_0 /tier2flight/txn3/tier2_3/tier3_0 /tier3schedl/txn3/tier3_0 /ctauth/service1.asmx /ctsecure/service1.asmx /tradefast/securitiessearch.aspx /tradefast/scriptresource.axd /tradefast/webresource.axd /tradefast/orders.aspx /cttradinghouse/tradinghouseservice.svc

2) Create a transaction type in the APM interface (see “Creating a Simple Transaction Type from Scratch“) to use as a template. For example, the template might look like this: "###################" "name","" "appliesTo","endtoend_and_local" "financialImpact","" "description","Top URL path as determined by | transactioncount -group_by urlpath" "criteria","url" "match","any" "pattern","**"

3) Export the template (and any other existing transaction types) to get started. Edit the exported CSV file.

4) Create 10 copies of the template of the template transaction type in the CSV file. Replace the value with corresponding values from the transactioncount search results. For example: "###################" "name","tier1client" "appliesTo","endtoend_and_local" "financialImpact","" "description","Top URL path as determined by | transactioncount -group_by urlpath" "criteria","url" "match","any" "pattern","*/tier1client/txn3/tier1_2/tier2_3/tier3_0*"

5) Save and re-import the CSV file with the new transaction types.

Example: Deleting All Transaction Types with Import

You can import a file with a single transaction type to delete all other transaction types. This is useful if you want to delete all transaction types in your environment. (Otherwise you must click the delete icon ( ) for each transaction type in the “Transaction Types Screen“.) Follow steps similar to these:

1) Optional: back up existing transaction types by clicking the Export button ( ) at the bottom of the list of transactions.

2) Create a CSV file similar to the following. It defines a single dummy transaction type you import and then delete:

Aternity APM Version 11.4.2 153 Transaction Types

"version","v2.0" "###################" "name","Delete me" "precedence","1" "isDefault","false" "appliesTo","endtoend_and_local" "financialImpact","0.0" "description","Delete me to get rid of the last TT." "criteria","url" "match","any" "pattern","*"

3) Click the Import button ( ) at the bottom of the list of transactions and specify the CSV file you created.

4) Click the delete icon ( ) for the single remaining Delete me transaction type.

Transaction Types Screen

This screen lists transaction types that have been defined. Use this screen to add, edit, or delete transaction types, and change their precedence. The Aternity icon ( ) next to a name in the list denotes one of the “Default Transaction Types“.

Add a Transaction Type

To add a new transaction type, click Define a Transaction Type. The “Define a Transaction Type Screen“ opens.

List Operations and Controls

The list of existing transaction types provides the following operations:

 Display details: Click the row for an existing transaction type to select it. Details for that transaction type appear in the Transaction Type Summary area below the list.

 Edit: To edit an existing transaction type, click the edit icon ( ) in the row for the transaction type you want to edit. The “Define a Transaction Type Screen“ opens with the settings for that transaction type.

 Delete: To delete an existing transaction type, click the delete icon ( ) in the row for the transaction type you want to delete. When you delete a transaction type, any “Transactional Thresholds“ that refer to it are also deleted. When you delete a transaction type, you receive a warning if any application definitions refer to it. To delete many transaction types at once, follow the steps described in “Example: Deleting All Transaction Types with Import“.

 Change precedence: To change the precedence of a transaction type, click the number in the Precedence column in the row for the transaction type. See “Transaction Type Precedence“ for details.

154 Aternity APM Version 11.4.2 Transaction Types

 Sort: To sort the list, click on any column heading to sort the column it in ascending order based on the column value. Click the column heading again to sort in descending order. An icon indicates ascending or descending sort order.

Transaction Type Precedence

Transaction data (scoped as specified in the “Applies To“ setting) can match only a single transaction type. APM evaluates transaction types in the order specified in the Precedence column (precedence 1 is evaluated first, 2 second, and so on). APM assigns the first transaction type that matches a specific transaction and ignores any subsequent transaction types that match. As described in “Default Transaction Types“, APM supplies a set of transaction types that create blocks of precedence values. When you create a new transaction type in the “Define a Transaction Type Screen“, APM assigns the next available precedence above 999. Change the precedence of a transaction type by selecting its row and clicking the Precedence number. Supply a valid precedence value (from 1 to 9,999) in the text box that opens. APM assigns the new precedence value to that transaction type. It shifts the other precedence values as appropriate. For example, if you have 10 transaction types and change the precedence of the transaction type currently assigned precedence 5 to be 8, then Transaction Types with precedence from 6 to 8 will be adjusted to 5 to 7.

Add Defaults

The Add Defaults button ( ) adds “Default Transaction Types“. This button appears only after upgrades from versions earlier than Version 10.12.0, and only when there are existing transaction types. See “Adding Default Transaction Types“ for more details.

Import and Export

To export all the transaction type definitions to a file that you can edit, click the Export button ( ). To import the edited transaction types, click the Import button ( ). See “Bulk Editing Transaction Types with Export and Import“ for more details.

Download default Transaction Types as a CSV file (requires internet access)

The Download default Transaction Types as a CSV file (requires internet access) link downloads the latest default transaction types from a Aternity server. See “Adding Default Transaction Types“ for more details.

Financial Impact by Transaction Type

Use these settings to quantify the financial impact of transactions. They affect the “Performance Graph Card“ when users select the “Show Financial Impact“' option.

Aternity APM Version 11.4.2 155 Transaction Types

Financial Impact Per Minute

This setting specifies a default financial impact value. It applies to:

 Transactions that match a transaction type that does not specify a value in the “Financial Impact Per Minute“ setting

 All transactions that do not match a transaction type For these transactions, the performance graph uses the Financial Impact Per Minute value to change the scale of the graph when users select Show Financial Impact. The default value of zero means that applicable transactions will not be used in the financial impact view. For the greatest flexibility, specify the Financial Impact Per Minute value for individual transaction types. For details on how the performance graph calculates financial impact, see the “Financial Impact Per Minute“ setting in the “Define a Transaction Type Screen“.)

Currency Unit

This setting specifies the currency unit to display in the performance graph when users select Show Financial Impact. The graph displays the value supplied here in the legend and tooltips. For example, if you supply złoty as the Currency Unit:

This is only a display value and does not affect the financial impact calculation.

Define a Transaction Type Screen

This section describes the fields that define a transaction type.

156 Aternity APM Version 11.4.2 Transaction Types

Name

A name that identifies the transaction type throughout the user interface. The name must be unique among transaction types. The names are case-insensitive, so you cannot have two transaction types named “FOO” and “foo”.

Note: Cannot Rename Transaction Types Try to think of a useful name before you save the transaction type. The name appears prominently in the user interface, and you cannot change the name once you save the transaction type.

Applies To

The scope of transaction trace data that this transaction type will be evaluated against. As noted in “Match Field List“, not all criteria are allowed with End-to-End Transactions and Transaction Segments and Transaction Segments. See “What are “End-to-End Transactions” and “Transaction Segments”?“ for more details.

Financial Impact Per Minute

This setting quantifies the financial impact of transactions that match the transaction type. It affects the “Performance Graph Card“ when users select the “Show Financial Impact“ option. When that option is selected, the performance graph uses the Financial Impact Per Minute value to change the vertical scale of the graph. Use the Financial Impact Per Minute value to assign a relative financial impact weight to different transaction types. For example, consider an investment firm call center where entry-level personnel run transactions that match the Call Center transaction type. In the same firm, automated trading software runs transactions of type Trades. The financial impact when transactions of type Call Center take a long time to run is much less than when transactions of type Trades take a long time. In this scenario, you would provide a small Financial Impact Per Minute value to transaction types Call Center and a large value for Trades. With a non-zero Financial Impact Per Minute value, instead of showing processing time in seconds, the graph shows the financial impact, calculated as follows: (processing time) / 60 * Financial Impact Per Minute

In the previous example, you might assign a Financial Impact Per Minute value of 5 to the Call Center transaction type and 100 to Trades. In the case where transactions of both types each happen consume 180 seconds, the financial impact would be 15 for Call Center and 300 for Trades. You can specify a currency label with the “Currency Unit“ setting in the “Transaction Types Screen“. Set the value to zero if you do not want the performance graph to calculate the financial impact for transactions that match the transaction type. (This can be useful to omit a transaction type from the default financial impact calculation as specified in the “Financial Impact Per Minute“ value in the “Transaction Types Screen“.) This setting is optional. If you leave it blank (the default), it uses the “Financial Impact Per Minute“ value on the Manage Transaction Types screen.

Aternity APM Version 11.4.2 157 Transaction Types

Description

An optional description that appears in the “Transaction Types Screen“.

Match the following criteria

This area specifies the patterns that you want this transaction type to match. Click “Add Criteria“ to specify one or more criteria that each specify patterns. Each criterion specifies a field to match (see “Match Field List“) and one or more corresponding patterns to match. By combining multiple criteria and, within a criterion, multiple patterns, you can create flexible sets of pattern matches using logical AND and OR operations:

 A transaction must match all criteria specified for a transaction type. (In other words, APM evaluates multiple criteria as a logical AND operation.)

 Within a criterion, you can choose whether the specified transaction field value matches any of the corresponding patterns (a logical OR operation) or must match all of them (a logical AND operation).

Add Criteria

When you add a new transaction, the screen opens with a single “Criterion Details“ area. Click Add Criteria to add more criteria.

Criterion Details

For each criterion, you must specify a transaction field and one or more patterns you want its value to match. If you specify multiple patterns, you also need to specify whether the field value matches any or all of them.

Match any / Match all

This setting has an effect only if you specify multiple patterns in a criterion:

 Choose any if the field value needs to match only one of the patterns (a logical OR operation).

 Choose all if the field value must match all of the patterns (a logical AND operation).

158 Aternity APM Version 11.4.2 Transaction Types

Match Field List

Values in the Match any/all of these list are similar to search fields described in “Search Field Reference“.

Match Field Matches Search Field

Categories Category of class (such as EJB, JDBC, or Servlet for “category“ Java; ADO or Web for .NET) called during the transaction. Class.method.parameters Class, method, parameter name, and parameter “class.method.parameter“ value of a call during the transaction. “ejb.param“ “parameter“ Use this match field for any method parameter “parametervalue“ name and value, including those any matched by “rmi.param“ the method-parameter search fields shown in the “servlet.param“ next column. “web.param“ “webservice.param“ Class.methods Class and method name of a call. “class.method“

Classes Class name of a call during the transaction. “class“

HTTP Headers An arbitrary header value in an HTTP request “header“ within the transaction.

instances The name of a .NET app domain or Java JVM that “instance“ APM is monitoring. APM discovers processes automatically and lists them in the “Agent Details Screen“ where you can choose to “instrument” and monitor them. Page Tags Values of “Page Tags“ specified as custom fields in “pageload.time.backend“ the JavaScript snippet that APM uses to monitor web page performance. Specify page tags in the “Customize Snippet“ settings of the “Define a Configuration Screen“. Specifying Page Tags is not allowed with “Applies To“ settings of either End-to-End Transactions and Transaction Segments or Transaction Segments. See “What are “End-to-End Transactions” and “Transaction Segments”?“ for details.

Page Titles The title of the web page as specified by the HTML “pagetitle“ tag in the page. Specifying Page Titles is not allowed with “Applies To“ settings of either End-to-End Transactions and Transaction Segments or Transaction Segments. See “What are “End-to-End Transactions” and “Transaction Segments”?“ for details. Process Users User name of the operating system user that “processuser“ launched the application process. Aternity APM Version 11.4.2 159 Transaction TypesMatch Field Matches Search FieldServers The system name of a system running the APM “server“ agent software that generated the trace data. Specifying Servers is not allowed with the following combination of settings: • “Applies To“ setting of either End-to-End Transactions and Transaction Segments or Transaction Segments • “Match any / Match all“ setting of all •Multiple “Pattern to Match“ specifications See “What are “End-to-End Transactions” and “Transaction Segments”?“ for details. SQL Families An SQL statement group executed by a method call “sqlfamily“ in the application. SQL An SQL statement or procedure executed as part of “sql“ Statements/Procedures a JDBC or ADO.NET call. URL Outbounds “Outbound” URLs in transactions. “urloutbound“URL Parameters A parameter in the query string of an “inbound” “urlparameter“ URL requested by another tier. URLs “Inbound” URLs in transactions. “url“Web Service Methods The name of a web service method called during “webservicemethod“ the transaction. 160 Aternity APM Version 11.4.2 Transaction TypesPattern to MatchSpecify the pattern you want the value of the field to match. To specify multiple patterns that the field value must match, click the button.  Enter a space to see values for existing fields. You can accept any value and modify it with wildcards to suit your purposes. For example:  Patterns are not case sensitive. Patterns can use an asterisk (*) as a wildcard to match zero or more characters and a question mark (?) to match a single character. Use a backslash (\) as an escape character before either wildcard to match that character in a string. You can use wildcards anywhere within the pattern. For example, the following pattern will match systems named vh6-rh5-4 and vh8-w2k8-1, but not vh14-w2k8-3 or ttw1: vh?-*-?Aternity APM Version 11.4.2 161 Transaction Types162 Aternity APM Version 11.4.2 CHAPTER 20 IP Geography Mappings ScreenNote: This Screen Not Available to Read-Only Users Configuration screens are available only when you log in to APM using an administrative account. See the “User Roles“ topic for details on administrative and read-only accounts. APM tries to determine the geographic location of users that are accessing monitored applications from the IP address of the user’s system. The analysis server gets the user IP address from web page load data, when it is available, and from transaction trace data if web page load data is not available. The analysis server detects the location of many IP addresses automatically. However, for some IP addresses (secondary addresses, for example), the geographic location is reported as unknown or other. If you know the location for specific IP addresses, you can override the default location detection process and manually map a custom location with any address. Use this screen to create custom mappings. The IP Geography Mappings screen lists any existing mappings. To create new mappings, click Add and IP geography mapping icon. The “Use secondary IPs to locate users (advanced)“ setting may help map IP addresses in some environments.Add IP Geography Mapping Dialog To add a custom mapping, supply values for the “IP Range“, “Country“, and “Region“ fields. Other fields are optional. IP RangeAn IP address or range of addresses, using IPv4 dot-decimal notation. (IPv6 notation is not supported.) You can provide the address in any of the following forms: A single IP address. For example: 10.46.35.253 A range of IP addresses specified using asterisks (*) as wildcard characters. You can supply a wildcard for any octet. Once you use a wildcard, remaining octets (to the right) must also specify a wildcard. For example:Aternity APM Version 11.4.2 163 IP Geography Mappings Screen10.46.35.* 10.46.*.* 10.*.*.* A range of IP addresses separated by a hyphen. For example: 10.46.35.0-10.46.35.255 A range of IP addresses specified using CIDR (Classless Inter-Domain Routing) notation (external link to more details). For example, here is the CIDR representation of the range 10.46.35.0-10.46.35.255: 10.46.35.0/24 Avoid creating entries with identical IP addresses or ranges. It is unpredictable which entry will take effect. If custom mappings specify overlapping IP ranges, APM tries to assign an address to the mapping that most closely matches. The order of the mappings does not matter. The following examples show how this works.With These Overlapping Mapping Entries This Address Maps to10.46.35.0/24 Italy 10.46.35.25 Italy 10.46.0.0/16 Algeria 10.46.35.88 10.46.35.92 10.46.40.104 Algeria10.46.*.* Greenland 10.46.35.25 Brazil 10.46.35.* Brazil 10.46.35.88 Australia 10.46.35.90-10.46.35.95 Denmark 10.46.35.92 Denmark 10.46.35.80-10.46.35.95 Australia 10.46.40.104 Greenland CountryThe country to associate with the value in the IP Range setting. Choose a value from the list.RegionThe region within a country to associate with the value in the IP Range setting. Choose a value from the list. Lat/LongThe latitude and longitude of the user’s location. This setting is optional and has no effect on the country and region that APM associates with a particular IP address. Its only use is to provide coordinates to Google maps. Supplying coordinates here causes the icon to appear in the Location line of “Overview Tab“ block of the “Transaction Details“ screen. Users click the icon to open Google maps in a separate browser window with a map that is centered on the coordinates. Specify values in decimal format (90 to -90 for latitude, 180 to -180 for longitude). For example, for Nashua, New Hampshire, USA: Lat 42.765366 Long -71.467566 You can find coordinates for any location from web sites such as http://www.latlong.net.164 Aternity APM Version 11.4.2 IP Geography Mappings ScreenDescriptionAn optional description that appears in the list of custom mappings.Bulk EditClick Bulk Edit to open an editing window that shows any existing custom mapping entries as comma-separated-values (CSV) records. You can directly edit entries in the window or copy the contents of entries in another file into the window. For example: "IP Range","Country Code","Region Code","Latitude","Longitude","Description" "10.46.40.104","br","27","-23.55052","-46.633309","nhx1-w2k8r2-12"Do not change the first record with the column headings. The “List of Country and Region Codes“ shows valid values for the Country Code and Region Code fields in the CSV record.Use secondary IPs to locate users (advanced)This specialized setting can be useful to geographically identify the IP addresses of users in some environments (such as in a VPN or branch office environment). Ordinarily, APM first checks the HTTP request IP address (the value of the “user.ip“ search field) to identify the user’s location. Selecting this setting causes APM to instead first check the secondary IP address (the address obtained from the X-Forwarded-For (XFF) HTTP header and the value of the “user.ip.xff“ search field) in the request. Use this setting if you know the geographic locations that correspond to specific secondary IP addresses:1) Select this setting.2) Add a custom mapping between those IP addresses and their locations as described in “Add IP Geography Mapping Dialog“.List of Country and Region Codes The following table shows the country and region codes recognized in the Bulk Edit Geography Mappings window that opens when you click “Bulk Edit“. Country Code (Country) Region Code RegionAND (Andorra) 2 Canillo 3 Encamp 4 La Massana 5Ordino 6 Sant Julia de LoriaAternity APM Version 11.4.2 165 IP Geography Mappings ScreenCountry Code (Country) Region Code Region7 Andorra la Vella 8 Escaldes-Engordany ARE (United Arab Emirates) 1 Abu Dhabi 2Ajman 3Dubai 4 Fujairah 5 Ras Al Khaimah 6Sharjah 7Umm Al Quwain AFG (Afghanistan) 1 Badakhshan 2Badghis 3 Baghlan 5 Bamian 6 Farah 7 Faryab 8 Ghazni 9Ghowr 10 Helmand 11 Herat 13 Kabol 14 Kapisa 17 Lowgar 18 Nangarhar 19 Nimruz 23 Kandahar 24 Kondoz 26 Takhar 27 Vardak 28 Zabol 29 Paktika 30 Balkh 31 Jowzjan 32 Samangan 33 Sar-e Pol166 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region34 Konar 35 Laghman 36 Paktia 37 Khowst 38 Nurestan 39 Oruzgan 40 Parvan 41 Daykondi 42 Panjshir ATG (Antigua and Barbuda) 1 Barbuda 3Saint George 4 Saint John 5Saint Mary 6Saint Paul 7 Saint Peter 8 Saint Philip 9Redonda ALB (Albania) 40 Berat 41 Diber 42 Durres 43 Elbasan 44 Fier 45 Gjirokaster 46 Korce 47 Kukes 48 Lezhe 49 Shkoder 50 Tirane 51 Vlore ARM (Armenia) 1 Aragatsotn 2 Ararat 3Armavir 4Gegharkunik 5 KotaykAternity APM Version 11.4.2 167 IP Geography Mappings ScreenCountry Code (Country) Region Code Region6 Lorri 7Shirak 8Syunik 9Tavush 10 Vayots Dzor 11 Yerevan AGO (Angola) 1 Benguela 2Bie 3 Cabinda 4 Cuando Cubango 5 Cuanza Norte 6 Cuanza Sul 7Cunene 8 Huambo 9Huila 12 Malanje 13 Namibe 14 Moxico 15 Uige 16 Zaire 17 Lunda Norte 18 Lunda Sul 19 Bengo 20 Luanda ARG (Argentina) 1 Buenos Aires 2 Catamarca 3Chaco 4Chubut 5Cordoba 6 Corrientes 7 Distrito Federal 8Entre Rios 9Formosa 10 Jujuy168 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region11 La Pampa 12 La Rioja 13 Mendoza 14 Misiones 15 Neuquen 16 Rio Negro 17 Salta 18 San Juan 19 San Luis 20 Santa Cruz 21 Santa Fe 22 Santiago del Estero 23 Tierra del Fuego 24 Tucuman AUT (Austria) 1 Burgenland 2Karnten 3 Niederosterreich 4Oberosterreich 5 Salzburg 6Steiermark 7Tirol 8 Vorarlberg 9Wien AUS (Australia) 1 Australian Capital Territory 2New South Wales 3 Northern Territory 4 Queensland 5 South Australia 6Tasmania 7Victoria 8 Western Australia AZE (Azerbaijan) 1 Abseron 2 Agcabadi 3AgdamAternity APM Version 11.4.2 169 IP Geography Mappings ScreenCountry Code (Country) Region Code Region4Agdas 5Agstafa 6Agsu 7 Ali Bayramli 8 Astara 9 Baki 10 Balakan 11 Barda 12 Beylaqan 13 Bilasuvar 14 Cabrayil 15 Calilabad 16 Daskasan 17 Davaci 18 Fuzuli 19 Gadabay 20 Ganca 21 Goranboy 22 Goycay 23 Haciqabul 24 Imisli 25 Ismayilli 26 Kalbacar 27 Kurdamir 28 Lacin 29 Lankaran 30 Lankaran 31 Lerik 32 Masalli 33 Mingacevir 34 Naftalan 35 Naxcivan 36 Neftcala 37 Oguz170 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region38 Qabala 39 Qax 40 Qazax 41 Qobustan 42 Quba 43 Qubadli 44 Qusar 45 Saatli 46 Sabirabad 47 Saki 48 Saki 49 Salyan 50 Samaxi 51 Samkir 52 Samux 53 Siyazan 54 Sumqayit 55 Susa 56 Susa 57 Tartar 58 Tovuz 59 Ucar 60 Xacmaz 61 Xankandi 62 Xanlar 63 Xizi 64 Xocali 65 Xocavand 66 Yardimli 67 Yevlax 68 Yevlax 69 Zangilan 70 Zaqatala 71 ZardabAternity APM Version 11.4.2 171 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionBIH (Bosnia and Herzegovina) 1 Federation of Bosnia and Herzegovina 2 Republika SrpskaBRB (Barbados) 1 Christ Church 2Saint Andrew 3Saint George 4 Saint James 5 Saint John 6Saint Joseph 7Saint Lucy 8 Saint Michael 9 Saint Peter 10 Saint Philip 11 Saint Thomas BGD (Bangladesh) 81 Dhaka 82 Khulna 83 Rajshahi 84 Chittagong 85 Barisal 86 Sylhet BEL (Belgium) 1 Antwerpen 3Hainaut 4Liege 5Limburg 6Luxembourg 7Namur 8 Oost-Vlaanderen 9 West-Vlaanderen 10 Brabant Wallon 11 Brussels Hoofdstedelijk Gewest 12 Vlaams-Brabant 13 Flanders 14 Wallonia BFA (Burkina Faso) 15 Bam 19 Boulkiemde172 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region20 Ganzourgou 21 Gnagna 28 Kouritenga 33 Oudalan 34 Passore 36 Sanguie 40 Soum 42 Tapoa 44 Zoundweogo 45 Bale 46 Banwa 47 Bazega 48 Bougouriba 49 Boulgou 50 Gourma 51 Houet 52 Ioba 53 Kadiogo 54 Kenedougou 55 Komoe 56 Komondjari 57 Kompienga 58 Kossi 59 Koulpelogo 60 Kourweogo 61 Leraba 62 Loroum 63 Mouhoun 64 Namentenga 65 Naouri 66 Nayala 67 Noumbiel 68 Oubritenga 69 PoniAternity APM Version 11.4.2 173 IP Geography Mappings ScreenCountry Code (Country) Region Code Region70 Sanmatenga 71 Seno 72 Sissili 73 Sourou 74 Tuy 75 Yagha 76 Yatenga 77 Ziro 78 Zondoma BGR (Bulgaria) 33 Mikhaylovgrad 38 Blagoevgrad 39 Burgas 40 Dobrich 41 Gabrovo 42 Grad Sofiya 43 Khaskovo 44 Kurdzhali 45 Kyustendil 46 Lovech 47 Montana 48 Pazardzhik 49 Pernik 50 Pleven 51 Plovdiv 52 Razgrad 53 Ruse 54 Shumen 55 Silistra 56 Sliven 57 Smolyan 58 Sofiya 59 Stara Zagora 60 Turgovishte 61 Varna174 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region62 Veliko Turnovo 63 Vidin 64 Vratsa 65 Yambol BHR (Bahrain) 1 Al Hadd 2Al Manamah 5 Jidd Hafs 6Sitrah 8 Al Mintaqah al Gharbiyah 9 Mintaqat Juzur Hawar 10 Al Mintaqah ash Shamaliyah 11 Al Mintaqah al Wusta 12 Madinat 13 Ar Rifa 14 Madinat Hamad 15 Al Muharraq 16 Al Asimah 17 Al Janubiyah 18 Ash Shamaliyah 19 Al Wusta BDI (Burundi) 2 Bujumbura 9 Bubanza 10 Bururi 11 Cankuzo 12 Cibitoke 13 Gitega 14 Karuzi 15 Kayanza 16 Kirundo 17 Makamba 18 Muyinga 19 Ngozi 20 Rutana 21 RuyigiAternity APM Version 11.4.2 175 IP Geography Mappings ScreenCountry Code (Country) Region Code Region22 Muramvya 23 Mwaro BEN (Benin) 7 Alibori 8Atakora 9 Atlanyique 10 Borgou 11 Collines 12 Kouffo 13 Donga 14 Littoral 15 Mono 16 Oueme 17 Plateau 18 Zou BMU (Bermuda) 1 Devonshire 2 Hamilton 3 Hamilton 4Paget 5Pembroke 6Saint George 7Saint George’s 8 Sandys 9Smiths 10 Southampton 11 Warwick BRN (Brunei Darussalam) 7 Alibori 8 Belait 9 Brunei and Muara 10 Temburong 11 Collines 12 Kouffo 13 Donga 14 Littoral 15 Tutong176 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region16 Oueme 17 Plateau 18 Zou BOL (Bolivia) 1 Chuquisaca 2Cochabamba 3 El Beni 4 La Paz 5Oruro 6Pando 7 Potosi 8Santa Cruz 9Tarija BRA (Brazil) 1 Acre 2Alagoas 3Amapa 4 Amazonas 5Bahia 6Ceara 7 Distrito Federal 8 Espirito Santo 11 Mato Grosso do Sul 13 Maranhao 14 Mato Grosso 15 Minas Gerais 16 Para 17 Paraiba 18 Parana 20 Piaui 21 Rio de Janeiro 22 Rio Grande do Norte 23 Rio Grande do Sul 24 Rondonia 25 Roraima 26 Santa CatarinaAternity APM Version 11.4.2 177 IP Geography Mappings ScreenCountry Code (Country) Region Code Region27 Sao Paulo 28 Sergipe 29 Goias 30 Pernambuco 31 Tocantins BHS (Bahamas) 5 Bimini 6Cat Island 10 Exuma 13 Inagua 15 Long Island 16 Mayaguana 18 Ragged Island 22 Harbour Island 23 New Providence 24 Acklins and Crooked Islands 25 Freeport 26 Fresh Creek 27 Governor’s Harbour 28 Green Turtle Cay 29 High Rock 30 Kemps Bay 31 Marsh Harbour 32 Nichollstown and Berry Islands 33 Rock Sound 34 Sandy Point 35 San Salvador and Rum Cay BTN (Bhutan) 5 Bumthang 6 Chhukha 7 Chirang 8 Daga 9Geylegphug 10 Ha 11 Lhuntshi 12 Mongar178 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region13 Paro 14 Pemagatsel 15 Punakha 16 Samchi 17 Samdrup 18 Shemgang 19 Tashigang 20 Thimphu 21 Tongsa 22 Wangdi Phodrang BWA (Botswana) 1 Central 3 Ghanzi 4 Kgalagadi 5Kgatleng 6 Kweneng 8 North-East 9 South-East 10 Southern 11 North-West BLR (Belarus) 1 Brestskaya Voblast 2 Homyelskaya Voblasts 3 Hrodzyenskaya Voblast 4Minsk 5 Minskaya Voblasts 6 Mahilyowskaya Voblasts 7 Vitsyebskaya Voblasts BLZ (Belize) 1 Belize 2 Cayo 3 Corozal 4Orange Walk 5 Stann Creek 6Toledo CAN (Canada) AB Alberta BC British ColumbiaAternity APM Version 11.4.2 179 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionMB Manitoba NB New Brunswick NL Newfoundland NS Nova Scotia NT Northwest Territories NU Nunavut ON Ontario PE Prince Edward Island QC Quebec SK Saskatchewan YT Yukon Territory COD (The Democratic Republic 1 Bandundu of the Congo)2 Equateur 4 Kasai-Oriental 5 Katanga 6 Kinshasa 8Bas-Congo 9Orientale 10 Maniema 11 Nord-Kivu 12 Sud-Kivu CAF (Central African Republic) 1 Bamingui-Bangoran 2 Basse-Kotto 3 Haute-Kotto 4 Mambere-Kadei 5 Haut-Mbomou 6Kemo 7 Lobaye 8Mbomou 9 Nana-Mambere 11 Ouaka 12 Ouham 13 Ouham-Pende 14 Cuvette-Ouest180 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region15 Nana-Grebizi 16 Sangha-Mbaere 17 Ombella-Mpoko 18 Bangui COG (Congo) 1 Bouenza 4 Kouilou 5Lekoumou 6 Likouala 7Niari 8 Plateaux 10 Sangha 11 Pool 12 Brazzaville 13 Cuvette 14 Cuvette-Ouest CHE (Switzerland) 1 Aargau 2Ausser-Rhoden 3 Basel-Landschaft 4Basel-Stadt 5Bern 6Fribourg 7 Geneve 8Glarus 9 Graubunden 10 Inner-Rhoden 11 Luzern 12 Neuchatel 13 Nidwalden 14 Obwalden 15 Sankt Gallen 16 Schaffhausen 17 Schwyz 18 Solothurn 19 ThurgauAternity APM Version 11.4.2 181 IP Geography Mappings ScreenCountry Code (Country) Region Code Region20 Ticino 21 Uri 22 Valais 23 Vaud 24 Zug 25 Zurich 26 Jura CIV (Cote D’Ivorie) 74 Agneby 75 Bafing 76 Bas-Sassandra 77 Denguele 78 Dix-Huit Montagnes 79 Fromager 80 Haut-Sassandra 81 Lacs 82 Lagunes 83 Marahoue 84 Moyen-Cavally 85 Moyen-Comoe 86 Nazi-Comoe 87 Savanes 88 Sud-Bandama 89 Sud-Comoe 90 Vallee du Bandama 91 Worodougou 92 Zanzan CHL (Chile) 1 Valparaiso 2 Aisen del General Carlos Ibanez del Campo 3 Antofagasta 4Araucania 5Atacama 6Bio-Bio 7 Coquimbo182 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region8 Libertador General Bernardo OHiggins 9Los Lagos10 Magallanes y de la Antartica Chilena 11 Maule 12 Region Metropolitana 13 Tarapaca 14 Los Lagos 15 Tarapaca 16 Arica y Parinacota 17 Los Rios CMR (Cameroon) 4 Est 5Littoral 7Nord-Ouest 8Ouest 9Sud-Ouest 10 Adamaoua 11 Centre 12 Extreme-Nord 13 Nord 14 Sud CHN (China) 1 Anhui 2 Zhejiang 3Jiangxi 4Jiangsu 5 Jilin 6Qinghai 7Fujian 8 Heilongjiang 9Henan10 Hebei 11 Hunan 12 Hubei 13 XinjiangAternity APM Version 11.4.2 183 IP Geography Mappings ScreenCountry Code (Country) Region Code Region14 Xizang 15 Gansu 16 Guangxi 18 Guizhou 19 Liaoning 20 Nei Mongol 21 Ningxia 22 Beijing 23 Shanghai 24 Shanxi 25 Shandong 26 Shaanxi 28 Tianjin 29 Yunnan 30 Guangdong 31 Hainan 32 Sichuan 33 Chongqing COL (Columbia) 1 Amazonas 2Antioquia 3Arauca 4Atlantico 8 Caqueta 9Cauca 10 Cesar 11 Choco 12 Cordoba 14 Guaviare 15 Guainia 16 Huila 17 La Guajira 19 Meta 20 Narino 21 Norte de Santander184 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region22 Putumayo 23 Quindio 24 Risaralda 25 San Andres y Providencia 26 Santander 27 Sucre 28 Tolima 29 Valle del Cauca 30 Vaupes 31 Vichada 32 Casanare 33 Cundinamarca 34 Distrito Especial 35 Bolivar 36 Boyaca 37 Caldas 38 Magdalena CRI (Costa Rica) 1 Alajuela 2Cartago 3 Guanacaste 4 Heredia 6Limon 7Puntarenas 8San Jose CUB (Cuba) 1 Pinar del Rio 2 Ciudad de la Habana 3 Matanzas 4 Isla de la Juventud 5 Camaguey 7 Ciego de Avila 8Cienfuegos 9Granma 10 Guantanamo 11 La HabanaAternity APM Version 11.4.2 185 IP Geography Mappings ScreenCountry Code (Country) Region Code Region12 Holguin 13 Las Tunas 14 Sancti Spiritus 15 Santiago de Cuba 16 Villa Clara CPV (Cape Verde) 1 Boa Vista 2Brava 4 Maio 5 Paul 7Ribeira Grande 8Sal 10 Sao Nicolau 11 Sao Vicente 13 Mosteiros 14 Praia 15 Santa Catarina 16 Santa Cruz 17 Sao Domingos 18 Sao Filipe 19 Sao Miguel 20 Tarrafal CYP (Cyprus) 1 Famagusta 2Kyrenia 3Larnaca 4 Nicosia 5 Limassol 6 Paphos CZE (Czech Republic) 52 Hlavni mesto Praha 78 Jihomoravsky kraj 79 Jihocesky kraj 80 Vysocina 81 Karlovarsky kraj 82 Kralovehradecky kraj 83 Liberecky kraj186 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region84 Olomoucky kraj 85 Moravskoslezsky kraj 86 Pardubicky kraj 87 Plzensky kraj 88 Stredocesky kraj 89 Ustecky kraj 90 Zlinsky kraj DEU (Germany) 1 Baden-Wurttemberg 2 Bayern 3Bremen 4Hamburg 5Hessen 6 Niedersachsen 7 Nordrhein-Westfalen 8 Rheinland-Pfalz 9 Saarland 10 Schleswig-Holstein 11 Brandenburg 12 Mecklenburg-Vorpommern 13 Sachsen 14 Sachsen-Anhalt 15 Thuringen 16 Berlin DJI (Djibouti) 1 Ali Sabieh 4Obock 5Tadjoura 6Dikhil 7Djibouti 8Arta DNK (Denmark) 17 Hovedstaden 18 Midtjylland 19 Nordjylland 20 Sjelland 21 SyddanmarkAternity APM Version 11.4.2 187 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionDMA (Dominica) 2 Saint Andrew 3 Saint David 4Saint George 5 Saint John 6Saint Joseph 7Saint Luke 8Saint Mark 9 Saint Patrick 10 Saint Paul 11 Saint Peter DOM (Dominican Republic) 1 Azua 2Baoruco 3 Barahona 4 Dajabon 5 Distrito Nacional 6Duarte 8 Espaillat 9 Independencia 10 La Altagracia 11 Elias Pina 12 La Romana 14 Maria Trinidad Sanchez 15 Monte Cristi 16 Pedernales 17 Peravia 18 Puerto Plata 19 Salcedo 20 Samana 21 Sanchez Ramirez 23 San Juan 24 San Pedro De Macoris 25 Santiago 26 Santiago Rodriguez 27 Valverde188 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region28 El Seibo 29 Hato Mayor 30 La Vega 31 Monsenor Nouel 32 Monte Plata 33 San Cristobal 34 Distrito Nacional 35 Peravia 36 San Jose de Ocoa 37 Santo Domingo DZA (Algeria) 1 Alger 3Batna 4 Constantine 6Medea 7Mostaganem 9Oran 10 Saida 12 Setif 13 Tiaret 14 Tizi Ouzou 15 Tlemcen 18 Bejaia 19 Biskra 20 Blida 21 Bouira 22 Djelfa 23 Guelma 24 Jijel 25 Laghouat 26 Mascara 27 Masila 29 Oum el Bouaghi 30 Sidi Bel Abbes 31 SkikdaAternity APM Version 11.4.2 189 IP Geography Mappings ScreenCountry Code (Country) Region Code Region33 Tebessa 34 Adrar 35 Ain Defla 36 Ain Temouchent 37 Annaba 38 Bechar 39 Bordj Bou Arreridj 40 Boumerdes 41 Chlef 42 El Bayadh 43 El Oued 44 El Tarf 45 Ghardaia 46 Illizi 47 Khenchela 48 Mila 49 Naama 50 Ouargla 51 Relizane 52 Souk Ahras 53 Tamanghasset 54 Tindouf 55 Tipaza 56 Tissemsilt ECU (Ecuador) 1 Galapagos 2 Azuay 3Bolivar 4 Canar 5 Carchi 6 Chimborazo 7 Cotopaxi 8El Oro 9 Esmeraldas 10 Guayas190 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region11 Imbabura 12 Loja 13 Los Rios 14 Manabi 15 Morona-Santiago 17 Pastaza 18 Pichincha 19 Tungurahua 20 Zamora-Chinchipe 22 Sucumbios 23 Napo 24 Orellana EST (Estonia) 1 Harjumaa 2Hiiumaa 3 Ida-Virumaa 4 Jarvamaa 5 Jogevamaa 6 Kohtla-Jarve 7 Laanemaa 8 Laane-Virumaa 9Narva 10 Parnu 11 Parnumaa 12 Polvamaa 13 Raplamaa 14 Saaremaa 15 Sillamae 16 Tallinn 17 Tartu 18 Tartumaa 19 Valgamaa 20 Viljandimaa 21 Vorumaa EGY (Egypt) 1 Ad DaqahliyahAternity APM Version 11.4.2 191 IP Geography Mappings ScreenCountry Code (Country) Region Code Region2 Al Bahr al Ahmar 3 Al Buhayrah 4Al Fayyum 5 Al Gharbiyah 6 Al Iskandariyah 7 Al Ismailiyah 8 Al Jizah 9Al Minufiyah 10 Al Minya 11 Al Qahirah 12 Al Qalyubiyah 13 Al Wadi al Jadid 14 Ash Sharqiyah 15 As Suways 16 Aswan 17 Asyut 18 Bani Suwayf 19 Bur Said 20 Dumyat 21 Kafr ash Shaykh 22 Matruh 23 Qina 24 Suhaj 26 Janub Sina 27 Shamal Sina ERI (Eritrea) 1 Anseba 2Debub 3 Debubawi Keyih Bahri 4 Gash Barka 5 Maakel 6 Semenawi Keyih Bahri ESP (Spain) 7 Islas Baleares 27 La Rioja 29 Madrid192 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region31 Murcia 32 Navarra 34 Asturias 39 Cantabria 51 Andalucia 52 Aragon 53 Canarias 54 Castilla-La Mancha 55 Castilla y Leon 56 Catalonia 57 Extremadura 58 Galicia 59 Pais Vasco 60 Comunidad Valenciana ETH (Ethiopia) 44 Adis Abeba 45 Afar 46 Amara 47 Binshangul Gumuz 48 Dire Dawa 49 Gambela Hizboch 50 Hareri Hizb 51 Oromiya 52 Sumale 53 Tigray 54 YeDebub Biheroch Bihereseboch na Hizboch FIN (Finland) 1 Aland 6Lapland 8Oulu 13 Southern Finland 14 Eastern Finland 15 Western Finland FJI (Fiji) 1 Central 2Eastern 3 NorthernAternity APM Version 11.4.2 193 IP Geography Mappings ScreenCountry Code (Country) Region Code Region4Rotuma 5Western FSM (Micronesia) 1 Kosrae 2 Pohnpei 3Chuuk 4Yap FRA (France) 97 Aquitaine 98 Auvergne 99 Basse-Normandie A1 Bourgogne A2 Bretagne A3 Centre A4 Champagne-Ardenne A5 Corse A6 Franche-Comte A7 Haute-Normandie A8 Ile-de-France A9 Languedoc-Roussillon B1 Limousin B2 Lorraine B3 Midi-Pyrenees B4 Nord-Pas-de-Calais B5 Pays de la Loire B6 Picardie B7 Poitou-Charentes B8 Provence-Alpes-Cote d Azur B9 Rhone-Alpes C1 Alsace GAB (Gabon) 1 Estuaire 2 Haut-Ogooue 3 Moyen-Ogooue 4Ngounie 5Nyanga 6 Ogooue-Ivindo194 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region7 Ogooue-Lolo 8 Ogooue-Maritime 9Woleu-Ntem GBR (United Kingdom) A1 Barking and Dagenham A2 Barnet A3 Barnsley A4 Bath and North East Somerset A5 Bedfordshire A6 Bexley A7 Birmingham A8 Blackburn with Darwen A9 Blackpool B1 Bolton B2 Bournemouth B3 Bracknell Forest B4 Bradford B5 Brent B6 Brighton and Hove B7 Bristol B8 Bromley B9 Buckinghamshire C1 Bury C2 Calderdale C3 Cambridgeshire C4 Camden C5 Cheshire C6 Cornwall C7 Coventry C8 Croydon C9 Cumbria D1 Darlington D2 Derby D3 Derbyshire D4 DevonAternity APM Version 11.4.2 195 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionD5 Doncaster D6 Dorset D7 Dudley D8 Durham D9 Ealing E1 East Riding of Yorkshire E2 East Sussex E3 Enfield E4 Essex E5 Gateshead E6 Gloucestershire E7 Greenwich E8 Hackney E9 Halton F1 Hammersmith and Fulham F2 Hampshire F3 Haringey F4 Harrow F5 Hartlepool F6 Havering F7 Herefordshire F8 Hertford F9 Hillingdon G1 Hounslow G2 Isle of Wight G3 Islington G4 Kensington and Chelsea G5 Kent G6 Kingston upon Hull G7 Kingston upon Thames G8 Kirklees G9 Knowsley H1 Lambeth H2 Lancashire196 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionH3 Leeds H4 Leicester H5 Leicestershire H6 Lewisham H7 Lincolnshire H8 Liverpool H9 London I1 Luton I2 Manchester I3 Medway I4 Merton I5 Middlesbrough I6 Milton Keynes I7 Newcastle upon Tyne I8 Newham I9 Norfolk J1 Northamptonshire J2 North East Lincolnshire J3 North Lincolnshire J4 North Somerset J5 North Tyneside J6 Northumberland J7 North Yorkshire J8 Nottingham J9 Nottinghamshire K1 Oldham K2 Oxfordshire K3 Peterborough K4 Plymouth K5 Poole K6 Portsmouth K7 Reading K8 Redbridge K9 Redcar and ClevelandAternity APM Version 11.4.2 197 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionL1 Richmond upon Thames L2 Rochdale L3 Rotherham L4 Rutland L5 Salford L6 Shropshire L7 Sandwell L8 Sefton L9 Sheffield M1 Slough M2 Solihull M3 Somerset M4 Southampton M5 Southend-on-Sea M6 South Gloucestershire M7 South Tyneside M8 Southwark M9 Staffordshire N1 St. Helens N2 Stockport N3 Stockton-on-Tees N4 Stoke-on-Trent N5 Suffolk N6 Sunderland N7 Surrey N8 Sutton N9 Swindon O1 Tameside O2 Telford and Wrekin O3 Thurrock O4 Torbay O5 Tower Hamlets O6 Trafford O7 Wakefield198 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionO8 Walsall O9 Waltham Forest P1 Wandsworth P2 Warrington P3 Warwickshire P4 West Berkshire P5 Westminster P6 West Sussex P7 Wigan P8 Wiltshire P9 Windsor and Maidenhead Q1 Wirral Q2 Wokingham Q3 Wolverhampton Q4 Worcestershire Q5 York Q6 Antrim Q7 Ards Q8 Armagh Q9 Ballymena R1 Ballymoney R2 Banbridge R3 Belfast R4 Carrickfergus R5 Castlereagh R6 Coleraine R7 Cookstown R8 Craigavon R9 Down S1 Dungannon S2 Fermanagh S3 Larne S4 Limavady S5 LisburnAternity APM Version 11.4.2 199 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionS6 Derry S7 Magherafelt S8 Moyle S9 Newry and Mourne T1 Newtownabbey T2 North Down T3 Omagh T4 Strabane T5 Aberdeen City T6 Aberdeenshire T7 Angus T8 Argyll and Bute T9 Scottish Borders U1 Clackmannanshire U2 Dumfries and Galloway U3 Dundee City U4 East Ayrshire U5 East Dunbartonshire U6 East Lothian U7 East Renfrewshire U8 Edinburgh U9 Falkirk V1 Fife V2 Glasgow City V3 Highland V4 Inverclyde V5 Midlothian V6 Moray V7 North Ayrshire V8 North Lanarkshire V9 Orkney W1 Perth and Kinross W2 Renfrewshire W3 Shetland Islands200 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionW4 South Ayrshire W5 South Lanarkshire W6 Stirling W7 West Dunbartonshire W8 Eilean Siar W9 West Lothian X1 Isle of Anglesey X2 Blaenau Gwent X3 Bridgend X4 Caerphilly X5 Cardiff X6 Ceredigion X7 Carmarthenshire X8 Conwy X9 Denbighshire Y1 Flintshire Y2 Gwynedd Y3 Merthyr Tydfil Y4 Monmouthshire Y5 Neath Port Talbot Y6 Newport Y7 Pembrokeshire Y8 Powys Y9 Rhondda Cynon Taff Z1 Swansea Z2 Torfaen Z3 Vale of Glamorgan Z4 Wrexham Z5 Bedfordshire Z6 Central Bedfordshire Z7 Cheshire East Z8 Cheshire West and Chester Z9 Isles of Scilly GRD (Grenada) 1 Saint AndrewAternity APM Version 11.4.2 201 IP Geography Mappings ScreenCountry Code (Country) Region Code Region2 Saint David 3Saint George 4 Saint John 5Saint Mark 6 Saint Patrick GEO (<a href="/tags/Georgia_(country)/" rel="tag">Georgia</a>) 1 Abashis Raioni 2 <a href="/tags/Abkhazia/" rel="tag">Abkhazia</a> 3 Adigenis Raioni 4Ajaria 5 Akhalgoris Raioni 6 Akhalkalakis Raioni 7 Akhaltsikhis Raioni 8Akhmetis Raioni 9 Ambrolauris Raioni 10 Aspindzis Raioni 11 Baghdatis Raioni 12 Bolnisis Raioni 13 Borjomis Raioni 14 <a href="/tags/Chiatura/" rel="tag">Chiatura</a> 15 Chkhorotsqus Raioni 16 Chokhatauris Raioni 17 Dedoplistsqaros Raioni 18 Dmanisis Raioni 19 Dushetis Raioni 20 Gardabanis Raioni 21 Gori 22 Goris Raioni 23 Gurjaanis Raioni 24 Javis Raioni 25 Karelis Raioni 26 Kaspis Raioni 27 Kharagaulis Raioni 28 Khashuris Raioni 29 Khobis Raioni202 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region30 Khonis Raioni 31 <a href="/tags/Kutaisi/" rel="tag">Kutaisi</a> 32 Lagodekhis Raioni 33 Lanchkhutis Raioni 34 Lentekhis Raioni 35 Marneulis Raioni 36 Martvilis Raioni 37 Mestiis Raioni 38 Mtskhetis Raioni 39 Ninotsmindis Raioni 40 Onis Raioni 41 Ozurgetis Raioni 42 <a href="/tags/Poti/" rel="tag">Poti</a> 43 Qazbegis Raioni 44 Qvarlis Raioni 45 <a href="/tags/Rustavi/" rel="tag">Rustavi</a> 46 Sachkheris Raioni 47 Sagarejos Raioni 48 Samtrediis Raioni 49 Senakis Raioni 50 Sighnaghis Raioni 51 <a href="/tags/Tbilisi/" rel="tag">Tbilisi</a> 52 Telavis Raioni 53 Terjolis Raioni 54 Tetritsqaros Raioni 55 Tianetis Raioni 56 Tqibuli 57 Tsageris Raioni 58 Tsalenjikhis Raioni 59 Tsalkis Raioni 60 Tsqaltubo 61 Vanis Raioni 62 Zestaponis Raioni 63 <a href="/tags/Zugdidi/" rel="tag">Zugdidi</a>Aternity APM Version 11.4.2 203 IP Geography Mappings ScreenCountry Code (Country) Region Code Region64 Zugdidis Raioni GHA (Ghana) 1 Greater Accra 2Ashanti 3 Brong-Ahafo 4Central 5Eastern 6 Northern 8 Volta 9Western 10 Upper East 11 Upper West GRL (Greenland) 1 Nordgronland 2Ostgronland 3Vestgronland GMB (Gambia) 1 Banjul 2Lower River 3 Central River 4 Upper River 5Western 7 North Bank GIN (Guinea) 1 Beyla 2Boffa 3Boke 4Conakry 5 Dabola 6 Dalaba 7 Dinguiraye 9 Faranah 10 Forecariah 11 Fria 12 Gaoual 13 Gueckedou 15 Kerouane 16 Kindia204 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region17 Kissidougou 18 Koundara 19 Kouroussa 21 Macenta 22 Mali 23 Mamou 25 Pita 27 Telimele 28 Tougue 29 Yomou 30 Coyah 31 Dubreka 32 Kankan 33 Koubia 34 Labe 35 Lelouma 36 Lola 37 Mandiana 38 Nzerekore 39 Siguiri GNQ (Equatorial Guinea) 3 Annobon 4Bioko Norte 5Bioko Sur 6 Centro Sur 7Kie-Ntem 8Litoral 9 Wele-Nzas GRC (Greece) 1 Evros 2Rodhopi 3 Xanthi 4Drama 5 Serrai 6Kilkis 7PellaAternity APM Version 11.4.2 205 IP Geography Mappings ScreenCountry Code (Country) Region Code Region8Florina 9 Kastoria 10 Grevena 11 Kozani 12 Imathia 13 Thessaloniki 14 Kavala 15 Khalkidhiki 16 Pieria 17 Ioannina 18 Thesprotia 19 Preveza 20 Arta 21 Larisa 22 Trikala 23 Kardhitsa 24 Magnisia 25 Kerkira 26 Levkas 27 Kefallinia 28 Zakinthos 29 Fthiotis 30 Evritania 31 Aitolia kai Akarnania 32 Fokis 33 Voiotia 34 Evvoia 35 Attiki 36 Argolis 37 Korinthia 38 Akhaia 39 Ilia 40 Messinia 41 Arkadhia206 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region42 Lakonia 43 Khania 44 Rethimni 45 Iraklion 46 Lasithi 47 Dhodhekanisos 48 Samos 49 Kikladhes 50 Khios 51 Lesvos GTM (Guatemala) 1 Alta Verapaz 2 Baja Verapaz 3Chimaltenango 4Chiquimula 5 El Progreso 6Escuintla 7 Guatemala 8 Huehuetenango 9 Izabal 10 Jalapa 11 Jutiapa 12 Peten 13 Quetzaltenango 14 Quiche 15 Retalhuleu 16 Sacatepequez 17 San Marcos 18 Santa Rosa 19 Solola 20 Suchitepequez 21 Totonicapan 22 Zacapa GNB (Guinea-Bissau) 1 Bafata 2QuinaraAternity APM Version 11.4.2 207 IP Geography Mappings ScreenCountry Code (Country) Region Code Region4Oio 5 Bolama 6 Cacheu 7Tombali 10 Gabu 11 Bissau 12 Biombo GUY (Guyana) 10 Barima-Waini 11 Cuyuni-Mazaruni 12 Demerara-Mahaica 13 East Berbice-Corentyne 14 Essequibo Islands-West Demerara 15 Mahaica-Berbice 16 Pomeroon-Supenaam 17 Potaro-Siparuni 18 Upper Demerara-Berbice 19 Upper Takutu-Upper Essequibo HND (Honduras) 1 Atlantida 2Choluteca 3 Colon 4 Comayagua 5Copan 6 Cortes 7El Paraiso 8 Francisco Morazan 9 Gracias a Dios 10 Intibuca 11 Islas de la Bahia 12 La Paz 13 Lempira 14 Ocotepeque 15 Olancho 16 Santa Barbara 17 Valle208 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region18 Yoro HRV (Croatia) 1 Bjelovarsko-Bilogorska 2 Brodsko-Posavska 3 Dubrovacko-Neretvanska 4 Istarska 5 Karlovacka 6 Koprivnicko-Krizevacka 7Krapinsko-Zagorska 8Licko-Senjska 9Medimurska 10 Osjecko-Baranjska 11 Pozesko-Slavonska 12 Primorsko-Goranska 13 Sibensko-Kninska 14 Sisacko-Moslavacka 15 Splitsko-Dalmatinska 16 Varazdinska 17 Viroviticko-Podravska 18 Vukovarsko-Srijemska 19 Zadarska 20 Zagrebacka 21 Grad Zagreb HTI (Haiti) 3 Nord-Ouest 6Artibonite 7Centre 9Nord 10 Nord-Est 11 Ouest 12 Sud 13 Sud-Est 14 Grand Anse 15 Nippes HUN (Hungary) 1 Bacs-Kiskun 2 BaranyaAternity APM Version 11.4.2 209 IP Geography Mappings ScreenCountry Code (Country) Region Code Region3 Bekes 4 Borsod-Abauj-Zemplen 5 Budapest 6Csongrad 7Debrecen 8Fejer 9 Gyor-Moson-Sopron 10 Hajdu-Bihar 11 Heves 12 Komarom-Esztergom 13 Miskolc 14 Nograd 15 Pecs 16 Pest 17 Somogy 18 Szabolcs-Szatmar-Bereg 19 Szeged 20 Jasz-Nagykun-Szolnok 21 Tolna 22 Vas 23 Veszprem 24 Zala 25 Gyor 26 Bekescsaba 27 Dunaujvaros 28 Eger 29 Hodmezovasarhely 30 Kaposvar 31 Kecskemet 32 Nagykanizsa 33 Nyiregyhaza 34 Sopron 35 Szekesfehervar 36 Szolnok210 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region37 Szombathely 38 Tatabanya 39 Veszprem 40 Zalaegerszeg 41 Salgotarjan 42 Szekszard 43 Erd IDN (Indonesia) 1 Aceh 2 Bali 3Bengkulu 4 Jakarta Raya 5Jambi 7 Jawa Tengah 8 Jawa Timur 10 Yogyakarta 11 Kalimantan Barat 12 Kalimantan Selatan 13 Kalimantan Tengah 14 Kalimantan Timur 15 Lampung 17 Nusa Tenggara Barat 18 Nusa Tenggara Timur 21 Sulawesi Tengah 22 Sulawesi Tenggara 24 Sumatera Barat 26 Sumatera Utara 28 Maluku 29 Maluku Utara 30 Jawa Barat 31 Sulawesi Utara 32 Sumatera Selatan 33 Banten 34 Gorontalo 35 Kepulauan Bangka BelitungAternity APM Version 11.4.2 211 IP Geography Mappings ScreenCountry Code (Country) Region Code Region36 Papua 37 Riau 38 Sulawesi Selatan 39 Irian Jaya Barat 40 Kepulauan Riau 41 Sulawesi Barat IRL (Ireland) 1 Carlow 2 Cavan 3Clare 4Cork 6Donegal 7Dublin 10 Galway 11 Kerry 12 Kildare 13 Kilkenny 14 Leitrim 15 Laois 16 Limerick 18 Longford 19 Louth 20 Mayo 21 Meath 22 Monaghan 23 Offaly 24 Roscommon 25 Sligo 26 Tipperary 27 Waterford 29 Westmeath 30 Wexford 31 Wicklow ISR (Israel) 1 HaDarom 2 HaMerkaz212 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region3 HaZafon 4Hefa 5Tel Aviv 6Yerushalayim IND (India) 1 Andaman and Nicobar Islands 2 Andhra Pradesh 3Assam 5 Chandigarh 6 Dadra and Nagar Haveli 7Delhi 9 Gujarat 10 Haryana 11 Himachal Pradesh 12 Jammu and Kashmir 13 Kerala 14 Lakshadweep 16 Maharashtra 17 Manipur 18 Meghalaya 19 Karnataka 20 Nagaland 21 Orissa 22 Puducherry 23 Punjab 24 Rajasthan 25 Tamil Nadu 26 Tripura 28 West Bengal 29 Sikkim 30 Arunachal Pradesh 31 Mizoram 32 Daman and Diu 33 Goa 34 BiharAternity APM Version 11.4.2 213 IP Geography Mappings ScreenCountry Code (Country) Region Code Region35 Madhya Pradesh 36 Uttar Pradesh 37 Chhattisgarh 38 Jharkhand 39 Uttarakhand IRQ (Iraq) 1 Al Anbar 2 Al Basrah 3 Al Muthanna 4Al Qadisiyah 5 As Sulaymaniyah 6Babil 7 Baghdad 8Dahuk 9Dhi Qar 10 Diyala 11 Arbil 12 Karbala 13 At Tamim 14 Maysan 15 Ninawa 16 Wasit 17 An Najaf 18 Salah ad Din IRN (Iran) 1 Azarbayjan-e Bakhtari 3 Chahar Mahall va Bakhtiari 4 Sistan va Baluchestan 5 Kohkiluyeh va Buyer Ahmadi 7 Fars 8Gilan 9 Hamadan 10 Ilam 11 Hormozgan 12 Kerman 13 Bakhtaran214 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region15 Khuzestan 16 Kordestan 17 Mazandaran 18 Semnan Province 19 Markazi 21 Zanjan 22 Bushehr 23 Lorestan 24 Markazi 25 Semnan 26 Tehran 27 Zanjan 28 Esfahan 29 Kerman 30 Khorasan 31 Yazd 32 Ardabil 33 East Azarbaijan 34 Markazi 35 Mazandaran 36 Zanjan 37 Golestan 38 Qazvin 39 Qom 40 Yazd 41 Khorasan-e Janubi 42 Khorasan-e Razavi 43 Khorasan-e Shemali 44 Alborz ISL (Iceland) 3 Arnessysla 5 Austur-Hunavatnssysla 6 Austur-Skaftafellssysla 7Borgarfjardarsysla 9 EyjafjardarsyslaAternity APM Version 11.4.2 215 IP Geography Mappings ScreenCountry Code (Country) Region Code Region10 Gullbringusysla 15 Kjosarsysla 17 Myrasysla 20 Nordur-Mulasysla 21 Nordur-Tingeyjarsysla 23 Rangarvallasysla 28 Skagafjardarsysla 29 Snafellsnes- og Hnappadalssysla 30 Strandasysla 31 Sudur-Mulasysla 32 Sudur-Tingeyjarsysla 34 Vestur-Bardastrandarsysla 35 Vestur-Hunavatnssysla 36 Vestur-Isafjardarsysla 37 Vestur-Skaftafellssysla 38 Austurland 39 Hofuoborgarsvaoio 40 Norourland Eystra 41 Norourland Vestra 42 Suourland 43 Suournes 44 Vestfiroir 45 Vesturland ITA (Italy) 1 Abruzzi 2 Basilicata 3 Calabria 4 Campania 5 Emilia-Romagna 6 Friuli-Venezia Giulia 7 Lazio 8 Liguria 9Lombardia 10 Marche 11 Molise216 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region12 Piemonte 13 Puglia 14 Sardegna 15 Sicilia 16 Toscana 17 Trentino-Alto Adige 18 Umbria 19 Valle d Aosta 20 Veneto JAM (Jamaica) 1 Clarendon 2Hanover 4 Manchester 7 Portland 8Saint Andrew 9 Saint Ann 10 Saint Catherine 11 Saint Elizabeth 12 Saint James 13 Saint Mary 14 Saint Thomas 15 Trelawny 16 Westmoreland 17 Kingston JOR (Jordan) 2 Al Balqa 9Al Karak 12 At Tafilah 15 Al Mafraq 16 Amman 17 Az Zaraqa 18 Irbid 19 Maan 20 Ajlun 21 Al Aqabah 22 JarashAternity APM Version 11.4.2 217 IP Geography Mappings ScreenCountry Code (Country) Region Code Region23 Madaba JPN (Japan) 1 Aichi 2Akita 3 Aomori 4Chiba 5Ehime 6Fukui 7Fukuoka 8 Fukushima 9Gifu 10 Gumma 11 Hiroshima 12 Hokkaido 13 Hyogo 14 Ibaraki 15 Ishikawa 16 Iwate 17 Kagawa 18 Kagoshima 19 Kanagawa 20 Kochi 21 Kumamoto 22 Kyoto 23 Mie 24 Miyagi 25 Miyazaki 26 Nagano 27 Nagasaki 28 Nara 29 Niigata 30 Oita 31 Okayama 32 Osaka 33 Saga218 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region34 Saitama 35 Shiga 36 Shimane 37 Shizuoka 38 Tochigi 39 Tokushima 40 Tokyo 41 Tottori 42 Toyama 43 Wakayama 44 Yamagata 45 Yamaguchi 46 Yamanashi 47 Okinawa KEN (Kenya) 1 Central 2Coast 3Eastern 5Nairobi Area 6 North-Eastern 7Nyanza 8 Rift Valley 9Western KGZ (Kyrgyzstan) 1 Bishkek 2Chuy 3 Jalal-Abad 4Naryn 5Osh 6Talas 7 Ysyk-Kol 8Osh 9 Batken KHM (Cambodia) 1 Batdambang 2Kampong Cham 3 Kampong ChhnangAternity APM Version 11.4.2 219 IP Geography Mappings ScreenCountry Code (Country) Region Code Region4Kampong Speu 5Kampong Thum 6 Kampot 7 Kandal 8Koh Kong 9Kracheh 10 Mondulkiri 11 Phnum Penh 12 Pursat 13 Preah Vihear 14 Prey Veng 15 Ratanakiri Kiri 16 Siem Reap 17 Stung Treng 18 Svay Rieng 19 Takeo 25 Banteay Meanchey 29 Batdambang 30 Pailin KIR (Kiribati) 1 Gilbert Islands 2 Line Islands 3 Phoenix Islands COM (Comoros) 1 Anjouan 2Grande Comore 3Moheli KNA (Saint Kitts and Nevis) 1 Christ Church Nichola Town 2 Saint Anne Sandy Point 3 Saint George Basseterre 4 Saint George Gingerland 5 Saint James Windward 6 Saint John Capisterre 7 Saint John Figtree 8 Saint Mary Cayon 9Saint Paul Capisterre220 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region10 Saint Paul Charlestown 11 Saint Peter Basseterre 12 Saint Thomas Lowland 13 Saint Thomas Middle Island 15 Trinity Palmetto Point PRK (Korea (North)) 1 Chagang-do 3 Hamgyong-namdo 6 Hwanghae-namdo 7 Hwanghae-bukto 8 Kaesong-si 9 Kangwon-do 11 Pyongan-bukto 12 Pyongyang-si 13 Yanggang-do 14 Nampo-si 15 Pyongan-namdo 17 Hamgyong-bukto 18 Najin Sonbong-si KOR (Korea (South)) 1 Cheju-do 3 Cholla-bukto 5 Chungchong-bukto 6 Kangwon-do 10 Pusan-jikhalsi 11 Seoul-tukpyolsi 12 Inchon-jikhalsi 13 Kyonggi-do 14 Kyongsang-bukto 15 Taegu-jikhalsi 16 Cholla-namdo 17 Chungchong-namdo 18 Kwangju-jikhalsi 19 Taejon-jikhalsi 20 Kyongsang-namdo 21 Ulsan-gwangyoksiAternity APM Version 11.4.2 221 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionKWT (Kuwait) 1 Al Ahmadi 2Al Kuwayt 5Al Jahra 7 Al Farwaniyah 8 Hawalli 9 Mubarak al Kabir CYM (Kayman Islands) 1 Creek 2Eastern 3 Midland 4 South Town 5 Spot Bay 6Stake Bay 7West End 8Western KAZ (Kazakhstan) 1 Almaty 2Almaty City 3 Aqmola 4 Aqtobe 5 Astana 6Atyrau 7 West Kazakhstan 8 Bayqonyr 9 Mangghystau 10 South Kazakhstan 11 Pavlodar 12 Qaraghandy 13 Qostanay 14 Qyzylorda 15 East Kazakhstan 16 North Kazakhstan 17 Zhambyl LAO (Laos) 1 Attapu 2Champasak 3 Houaphan222 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region4Khammouan 5Louang Namtha 7Oudomxai 8 Phongsali 9 Saravan 10 Savannakhet 11 Vientiane 13 Xaignabouri 14 Xiangkhoang 17 Louangphrabang LBN (Lebanon) 1 Beqaa 2Al Janub 3Liban-Nord 4Beyrouth 5 Mont-Liban 6Liban-Sud 7 Nabatiye 8 Beqaa 9Liban-Nord 10 Aakk 11 Baalbek-Hermel LCA (Saint Lucia) 1 Anse-la-Raye 2Dauphin 3 Castries 4Choiseul 5Dennery 6Gros-Islet 7Laborie 8Micoud 9Soufriere 10 Vieux-Fort 11 Praslin LIE (Liechtenstein) 1 Balzers 2EschenAternity APM Version 11.4.2 223 IP Geography Mappings ScreenCountry Code (Country) Region Code Region3Gamprin 4 Mauren 5Planken 6Ruggell 7 Schaan 8Schellenberg 9Triesen 10 Triesenberg 11 Vaduz 21 Gbarpolu 22 River Gee LKA (Sri Lanka) 29 Central 30 North Central 32 North Western 33 Sabaragamuwa 34 Southern 35 Uva 36 Western 37 Eastern 38 Northern LBR (Liberia) 1 Bong 4 Grand Cape Mount 5Lofa 6 Maryland 7Monrovia 9Nimba 10 Sino 11 Grand Bassa 12 Grand Cape Mount 13 Maryland 14 Montserrado 17 Margibi 18 River Cess 19 Grand Gedeh224 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region20 Lofa 21 Gbarpolu 22 River Gee LSO (Lesotho) 10 Berea 11 Butha-Buthe 12 Leribe 13 Mafeteng 14 Maseru 15 Mohales Hoek 16 Mokhotlong 17 Qachas Nek 18 Quthing 19 Thaba-Tseka LTU (Lithuania) 56 Alytaus Apskritis 57 Kauno Apskritis 58 Klaipedos Apskritis 59 Marijampoles Apskritis 60 Panevezio Apskritis 61 Siauliu Apskritis 62 Taurages Apskritis 63 Telsiu Apskritis 64 Utenos Apskritis 65 Vilniaus Apskritis LUX (Luxembourg) 1 Diekirch 2 Grevenmacher 3Luxembourg LVA (Latvia) 1 Aizkraukles 2Aluksnes 3Balvu 4 Bauskas 5Cesu 6 Daugavpils 7 Daugavpils 8 DobelesAternity APM Version 11.4.2 225 IP Geography Mappings ScreenCountry Code (Country) Region Code Region9 Gulbenes 10 Jekabpils 11 Jelgava 12 Jelgavas 13 Jurmala 14 Kraslavas 15 Kuldigas 16 Liepaja 17 Liepajas 18 Limbazu 19 Ludzas 20 Madonas 21 Ogres 22 Preilu 23 Rezekne 24 Rezeknes 25 Riga 26 Rigas 27 Saldus 28 Talsu 29 Tukuma 30 Valkas 31 Valmieras 32 Ventspils 33 Ventspils LBY (Libya) 3 Al Aziziyah 5 Al Jufrah 8Al Kufrah 13 Ash Shati 30 Murzuq 34 Sabha 41 Tarhunah 42 Tubruq 45 Zlitan226 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region47 Ajdabiya 48 Al Fatih 49 Al Jabal al Akhdar 50 Al Khums 51 An Nuqat al Khams 52 Awbari 53 Az Zawiyah 54 Banghazi 55 Darnah 56 Ghadamis 57 Gharyan 58 Misratah 59 Sawfajjin 60 Surt 61 Tarabulus 62 Yafran MAR (Morocco) 45 Grand Casablanca 46 Fes-Boulemane 47 Marrakech-Tensift-Al Haouz 48 Meknes-Tafilalet 49 Rabat-Sale-Zemmour-Zaer 50 Chaouia-Ouardigha 51 Doukkala-Abda 52 Gharb-Chrarda-Beni Hssen 53 Guelmim-Es Smara 54 Oriental 55 Souss-Massa-Dr 56 Tadla-Azilal 57 Tanger-Tetouan 58 Taza-Al Hoceima-Taounate 59 La MCO (Monaco) 1 La Condamine 2 Monaco 3 Monte-CarloAternity APM Version 11.4.2 227 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionMDA (Moldova) 51 Gagauzia 57 Chisinau 58 Stinga Nistrului 59 Anenii Noi 60 Balti 61 Basarabeasca 62 Bender 63 Briceni 64 Cahul 65 Cantemir 66 Calarasi 67 Causeni 68 Cimislia 69 Criuleni 70 Donduseni 71 Drochia 72 Dubasari 73 Edinet 74 Falesti 75 Floresti 76 Glodeni 77 Hincesti 78 Ialoveni 79 Leova 80 Nisporeni 81 Ocnita 82 Orhei 83 Rezina 84 Riscani 85 Singerei 86 Soldanesti 87 Soroca 88 Stefan-Voda 89 Straseni228 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region90 Taraclia 91 Telenesti 92 Ungheni MDG (Madagascar) 1 Antsiranana 2 Fianarantsoa 3 Mahajanga 4Toamasina 5 Antananarivo 6 Toliara MKD (Macedonia) 1 Aracinovo 2Bac 3Belcista 4Berovo 5 Bistrica 6Bitola 7Blatec 8Bogdanci 9 Bogomila 10 Bogovinje 11 Bosilovo 12 Brvenica 13 Cair 14 Capari 15 Caska 16 Cegrane 17 Centar 18 Centar Zupa 19 Cesinovo 20 Cucer-Sandevo 21 Debar 22 Delcevo 23 Delogozdi 24 Demir Hisar 25 Demir KapijaAternity APM Version 11.4.2 229 IP Geography Mappings ScreenCountry Code (Country) Region Code Region26 Dobrusevo 27 Dolna Banjica 28 Dolneni 29 Dorce Petrov 30 Drugovo 31 Dzepciste 32 Gazi Baba 33 Gevgelija 34 Gostivar 35 Gradsko 36 Ilinden 37 Izvor 38 Jegunovce 39 Kamenjane 40 Karbinci 41 Karpos 42 Kavadarci 43 Kicevo 44 Kisela Voda 45 Klecevce 46 Kocani 47 Konce 48 Kondovo 49 Konopiste 50 Kosel 51 Kratovo 52 Kriva Palanka 53 Krivogastani 54 Krusevo 55 Kuklis 56 Kukurecani 57 Kumanovo 58 Labunista 59 Lipkovo230 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region60 Lozovo 61 Lukovo 62 Makedonska Kamenica 63 Makedonski Brod 64 Mavrovi Anovi 65 Meseista 66 Miravci 67 Mogila 68 Murtino 69 Negotino 70 Negotino-Polosko 71 Novaci 72 Novo Selo 73 Oblesevo 74 Ohrid 75 Orasac 76 Orizari 77 Oslomej 78 Pehcevo 79 Petrovec 80 Plasnica 81 Podares 82 Prilep 83 Probistip 84 Radovis 85 Rankovce 86 Resen 87 Rosoman 88 Rostusa 89 Samokov 90 Saraj 91 Sipkovica 92 Sopiste 93 SopotnicaAternity APM Version 11.4.2 231 IP Geography Mappings ScreenCountry Code (Country) Region Code Region94 Srbinovo 95 Staravina 96 Star Dojran 97 Staro Nagoricane 98 Stip 99 Struga A1 Strumica A2 Studenicani A3 Suto Orizari A4 Sveti Nikole A5 Tearce A6 Tetovo A7 Topolcani A8 Valandovo A9 Vasilevo B1 Veles B2 Velesta B3 Vevcani B4 Vinica B5 Vitoliste B6 Vranestica B7 Vrapciste B8 Vratnica B9 Vrutok C1 Zajas C2 Zelenikovo C3 Zelino C4 Zitose C5 Zletovo C6 Zrnovci MLI (Mali) 1 Bamako 3Kayes 4 Mopti 5Segou232 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region6Sikasso 7 Koulikoro 8 Tombouctou 9Gao 10 Kidal MMR (Myanmar) 1 Rakhine State 2 Chin State 3 Irrawaddy 4 Kachin State 5 Karan State 6 Kayah State 7Magwe 8 Mandalay 9Pegu 10 Sagaing 11 Shan State 12 Tenasserim 13 Mon State 14 Rangoon 17 Yangon MNG (Mongolia) 1 Arhangay 2 Bayanhongor 3 Bayan-Olgiy 5 Darhan 6Dornod 7Dornogovi 8Dundgovi 9 Dzavhan 10 Govi-Altay 11 Hentiy 12 Hovd 13 Hovsgol 14 Omnogovi 15 OvorhangayAternity APM Version 11.4.2 233 IP Geography Mappings ScreenCountry Code (Country) Region Code Region16 Selenge 17 Suhbaatar 18 Tov 19 Uvs 20 Ulaanbaatar 21 Bulgan 22 Erdenet 23 Darhan-Uul 24 Govisumber 25 Orhon MAC (Macau) 1 Ilhas 2 Macau MRT (Mauritania) 1 Hodh Ech Chargui 2Hodh El Gharbi 3 Assaba 4Gorgol 5Brakna 6 Trarza 7Adrar 8 Dakhlet Nouadhibou 9 Tagant 10 Guidimaka 11 Tiris Zemmour 12 Inchiri MSR (Montserrat) 1 Saint Anthony 2 Saint Georges 3 Saint Peter MUS (Mauritius) 12 Black River 13 Flacq 14 Grand Port 15 Moka 16 Pamplemousses 17 Plaines Wilhems 18 Port Louis234 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region19 Riviere du Rempart 20 Savanne 21 Agalega Islands 22 Cargados Carajos 23 Rodrigues MDV (Maldives) 1 Seenu 5 Laamu 30 Alifu 31 Baa 32 Dhaalu 33 Faafu 34 Gaafu Alifu 35 Gaafu Dhaalu 36 Haa Alifu 37 Haa Dhaalu 38 Kaafu 39 Lhaviyani 40 Maale 41 Meemu 42 Gnaviyani 43 Noonu 44 Raa 45 Shaviyani 46 Thaa 47 Vaavu MWI (Malawi) 2 Chikwawa 3Chiradzulu 4Chitipa 5Thyolo 6Dedza 7Dowa 8Karonga 9Kasungu 11 LilongweAternity APM Version 11.4.2 235 IP Geography Mappings ScreenCountry Code (Country) Region Code Region12 Mangochi 13 Mchinji 15 Mzimba 16 Ntcheu 17 Nkhata Bay 18 Nkhotakota 19 Nsanje 20 Ntchisi 21 Rumphi 22 Salima 23 Zomba 24 Blantyre 25 Mwanza 26 Balaka 27 Likoma 28 Machinga 29 Mulanje 30 Phalombe MEX (Mexico) 1 Aguascalientes 2 Baja California 3 Baja California Sur 4 Campeche 5Chiapas 6 Chihuahua 7 Coahuila de Zaragoza 8Colima 9 Distrito Federal 10 Durango 11 Guanajuato 12 Guerrero 13 Hidalgo 14 Jalisco 15 Mexico 16 Michoacan de Ocampo236 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region17 Morelos 18 Nayarit 19 Nuevo Leon 20 Oaxaca 21 Puebla 22 Queretaro de Arteaga 23 Quintana Roo 24 San Luis Potosi 25 Sinaloa 26 Sonora 27 Tabasco 28 Tamaulipas 29 Tlaxcala 30 Veracruz-Llave 31 Yucatan 32 Zacatecas MYS (Malaysia) 1 Johor 2Kedah 3 Kelantan 4 Melaka 5 Negeri Sembilan 6 Pahang 7Perak 8Perlis 9 Pulau Pinang 11 Sarawak 12 Selangor 13 Terengganu 14 Kuala Lumpur 15 Labuan 16 Sabah 17 Putrajaya MOZ (Mozambique) 1 Cabo Delgado 2 GazaAternity APM Version 11.4.2 237 IP Geography Mappings ScreenCountry Code (Country) Region Code Region3 Inhambane 4 Maputo 5 Sofala 6Nampula 7Niassa 8Tete 9 Zambezia 10 Manica 11 Maputo NAM (Namibia) 1 Bethanien 2 Caprivi Oos 3Boesmanland 4Gobabis 5 Grootfontein 6Kaokoland 7Karibib 8 Keetmanshoop 9Luderitz 10 Maltahohe 11 Okahandja 12 Omaruru 13 Otjiwarongo 14 Outjo 15 Owambo 16 Rehoboth 17 Swakopmund 18 Tsumeb 20 Karasburg 21 Windhoek 22 Damaraland 23 Hereroland Oos 24 Hereroland Wes 25 Kavango 26 Mariental238 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region27 Namaland 28 Caprivi 29 Erongo 30 Hardap 31 Karas 32 Kunene 33 Ohangwena 34 Okavango 35 Omaheke 36 Omusati 37 Oshana 38 Oshikoto 39 Otjozondjupa NER (Niger) 1 Agadez 2Diffa 3Dosso 4 Maradi 5Niamey 6Tahoua 7 Zinder 8Niamey NGA (Nigeria) 5 Lagos 11 Federal Capital Territory 16 Ogun 21 Akwa Ibom 22 Cross River 23 Kaduna 24 Katsina 25 Anambra 26 Benue 27 Borno 28 Imo 29 Kano 30 KwaraAternity APM Version 11.4.2 239 IP Geography Mappings ScreenCountry Code (Country) Region Code Region31 Niger 32 Oyo 35 Adamawa 36 Delta 37 Edo 39 Jigawa 40 Kebbi 41 Kogi 42 Osun 43 Taraba 44 Yobe 45 Abia 46 Bauchi 47 Enugu 48 Ondo 49 Plateau 50 Rivers 51 Sokoto 52 Bayelsa 53 Ebonyi 54 Ekiti 55 Gombe 56 Nassarawa 57 Zamfara NIC (Nicaragua) 1 Boaco 2 Carazo 3 Chinandega 4 Chontales 5Esteli 6Granada 7 Jinotega 8Leon 9 Madriz 10 Managua240 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region11 Masaya 12 Matagalpa 13 Nueva Segovia 14 Rio San Juan 15 Rivas 16 Zelaya 17 Autonoma Atlantico Norte 18 Region Autonoma Atlantico Sur NLD (Netherlands) 1 Drenthe 2Friesland 3 Gelderland 4 Groningen 5Limburg 6 Noord-Brabant 7 Noord-Holland 9Utrecht 10 Zeeland 11 Zuid-Holland 15 Overijssel 16 Flevoland NOR (Norway) 1 Akershus 2Aust-Agder 4Buskerud 5 Finnmark 6Hedmark 7Hordaland 8 More og Romsdal 9 Nordland 10 Nord-Trondelag 11 Oppland 12 Oslo 13 Ostfold 14 Rogaland 15 Sogn og FjordaneAternity APM Version 11.4.2 241 IP Geography Mappings ScreenCountry Code (Country) Region Code Region16 Sor-Trondelag 17 Telemark 18 Troms 19 Vest-Agder 20 Vestfold NPL (Nepal) 1 Bagmati 2Bheri 3Dhawalagiri 4Gandaki 5 Janakpur 6Karnali 7Kosi 8Lumbini 9 Mahakali 10 Mechi 11 Narayani 12 Rapti 13 Sagarmatha 14 Seti NRU (Nauru) 1 Aiwo 2Anabar 3Anetan 4Anibare 5Baiti 6Boe 7Buada 8 Denigomodu 9Ewa 10 Ijuw 11 Meneng 12 Nibok 13 Uaboe 14 Yaren NZL (New Zealand) 10 Chatham Islands242 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionE7 Auckland E8 Bay of Plenty E9 Canterbury F1 Gisborne F2 Hawkes Bay F3 Manawatu-Wanganui F4 Marlborough F5 Nelson F6 Northland F7 Otago F8 Southland F9 Taranaki G1 Waikato G2 Wellington G3 West Coast OMN (Oman) 1 Ad Dakhiliyah 2 Al Batinah 3Al Wusta 4 Ash Sharqiyah 5 Az Zahirah 6 Masqat 7 Musandam 8Zufar PAN (Panama) 1 Bocas del Toro 2 Chiriqui 3Cocle 4 Colon 5 Darien 6Herrera 7 Los Santos 8 Panama 9 San Blas 10 Veraguas PER (Peru) 1 AmazonasAternity APM Version 11.4.2 243 IP Geography Mappings ScreenCountry Code (Country) Region Code Region2 Ancash 3Apurimac 4 Arequipa 5 Ayacucho 6 Cajamarca 7 Callao 8Cusco 9 Huancavelica 10 Huanuco 11 Ica 12 Junin 13 La Libertad 14 Lambayeque 15 Lima 16 Loreto 17 Madre de Dios 18 Moquegua 19 Pasco 20 Piura 21 Puno 22 San Martin 23 Tacna 24 Tumbes 25 Ucayali PNG (Papua New Guinea) 1 Central 2 Gulf 3Milne Bay 4 Northern 5 Southern Highlands 6Western 7 North Solomons 8Chimbu 9 Eastern Highlands 10 East New Britain244 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region11 East Sepik 12 Madang 13 Manus 14 Morobe 15 New Ireland 16 Western Highlands 17 West New Britain 18 Sandaun 19 Enga 20 National Capital PHL (Philippines) 1 Abra 2 Agusan del Norte 3Agusan del Sur 4Aklan 5Albay 6Antique 7 Bataan 8 Batanes 9 Batangas 10 Benguet 11 Bohol 12 Bukidnon 13 Bulacan 14 Cagayan 15 Camarines Norte 16 Camarines Sur 17 Camiguin 18 Capiz 19 Catanduanes 20 Cavite 21 Cebu 22 Basilan 23 Eastern Samar 24 DavaoAternity APM Version 11.4.2 245 IP Geography Mappings ScreenCountry Code (Country) Region Code Region25 Davao del Sur 26 Davao Oriental 27 Ifugao 28 Ilocos Norte 29 Ilocos Sur 30 Iloilo 31 Isabela 32 Kalinga-Apayao 33 Laguna 34 Lanao del Norte 35 Lanao del Sur 36 La Union 37 Leyte 38 Marinduque 39 Masbate 40 Mindoro Occidental 41 Mindoro Oriental 42 Misamis Occidental 43 Misamis Oriental 44 Mountain 45 Negros Occidental 46 Negros Oriental 47 Nueva Ecija 48 Nueva Vizcaya 49 Palawan 50 Pampanga 51 Pangasinan 53 Rizal 54 Romblon 55 Samar 56 Maguindanao 57 North Cotabato 58 Sorsogon 59 Southern Leyte246 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region60 Sulu 61 Surigao del Norte 62 Surigao del Sur 63 Tarlac 64 Zambales 65 Zamboanga del Norte 66 Zamboanga del Sur 67 Northern Samar 68 Quirino 69 Siquijor 70 South Cotabato 71 Sultan Kudarat 72 Tawitawi A1 Angeles A2 Bacolod A3 Bago A4 Baguio A5 Bais A6 Basilan City A7 Batangas City A8 Butuan A9 Cabanatuan B1 Cadiz B2 Cagayan de Oro B3 Calbayog B4 Caloocan B5 Canlaon B6 Cavite City B7 Cebu City B8 Cotabato B9 Dagupan C1 Danao C2 Dapitan C3 Davao CityAternity APM Version 11.4.2 247 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionC4 Dipolog C5 Dumaguete C6 General Santos C7 Gingoog C8 Iligan C9 Iloilo City D1 Iriga D2 La Carlota D3 Laoag D4 Lapu-Lapu D5 Legaspi D6 Lipa D7 Lucena D8 Mandaue D9 Manila E1 Marawi E2 Naga E3 Olongapo E4 Ormoc E5 Oroquieta E6 Ozamis E7 Pagadian E8 Palayan E9 Pasay F1 Puerto Princesa F2 Quezon City F3 Roxas F4 San Carlos F5 San Carlos F6 San Jose F7 San Pablo F8 Silay F9 Surigao G1 Tacloban248 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionG2 Tagaytay G3 Tagbilaran G4 Tangub G5 Toledo G6 Trece Martires G7 Zamboanga G8 Aurora H2 Quezon H3 Negros Occidental I6 Compostela Valley I7 Davao del Norte J7 Kalinga K6 Malaybalay L9 Passi P1 Zambales M5 San Jose del Monte M6 San Juan M8 Santiago M9 Sarangani N1 Sipalay N3 Surigao del Norte P2 Zamboanga PAK (Pakistan) 1 Federally Administered Tribal Areas 2 Balochistan 3 North-West Frontier 4Punjab 5Sindh 6 Azad Kashmir 7 Northern Areas 8 Islamabad POL (Poland) 72 Dolnoslaskie 73 Kujawsko-Pomorskie 74 Lodzkie 75 LubelskieAternity APM Version 11.4.2 249 IP Geography Mappings ScreenCountry Code (Country) Region Code Region76 Lubuskie 77 Malopolskie 78 Mazowieckie 79 Opolskie 80 Podkarpackie 81 Podlaskie 82 Pomorskie 83 Slaskie 84 Swietokrzyskie 85 Warminsko-Mazurskie 86 Wielkopolskie 87 Zachodniopomorskie PSE (Palestine) GZ Gaza WE West Bank PRT (Portugal) 2 Aveiro 3Beja 4Braga 5Braganca 6 Castelo Branco 7Coimbra 8Evora 9 Faro 10 Madeira 11 Guarda 13 Leiria 14 Lisboa 16 Portalegre 17 Porto 18 Santarem 19 Setubal 20 Viana do Castelo 21 Vila Real 22 Viseu 23 Azores250 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionPRY (Paraguay) 1 Alto Parana 2Amambay 3Boqueron 4 Caaguazu 5 Caazapa 6Central 7Concepcion 8 Cordillera 10 Guaira 11 Itapua 12 Misiones 13 Neembucu 15 Paraguari 16 Presidente Hayes 17 San Pedro 19 Canindeyu 20 Chaco 21 Nueva Asuncion 23 Alto Paraguay QAT (Qatar) 1 Ad Dawhah 2 Al Ghuwariyah 3Al Jumaliyah 4 Al Khawr 5 Al Wakrah Municipality 6 Ar Rayyan 8 Madinat ach Shamal 9Umm Salal 10 Al Wakrah 11 Jariyan al Batnah 12 Umm Said ROU (Romania) 1 Alba 2Arad 3Arges 4 BacauAternity APM Version 11.4.2 251 IP Geography Mappings ScreenCountry Code (Country) Region Code Region5Bihor 6 Bistrita-Nasaud 7 Botosani 8Braila 9Brasov 10 Bucuresti 11 Buzau 12 Caras-Severin 13 Cluj 14 Constanta 15 Covasna 16 Dambovita 17 Dolj 18 Galati 19 Gorj 20 Harghita 21 Hunedoara 22 Ialomita 23 Iasi 25 Maramures 26 Mehedinti 27 Mures 28 Neamt 29 Olt 30 Prahova 31 Salaj 32 Satu Mare 33 Sibiu 34 Suceava 35 Teleorman 36 Timis 37 Tulcea 38 Vaslui 39 Valcea252 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region40 Vrancea 41 Calarasi 42 Giurgiu 43 Ilfov RUS (Russian Federation) 1 Adygeya 2 Aginsky Buryatsky AO 3Gorno-Altay 4 Altaisky krai 5Amur 6 Arkhangelsk 7Astrakhan 8 Bashkortostan 9Belgorod 10 Bryansk 11 Buryat 12 Chechnya 13 Chelyabinsk 14 Chita 15 Chukot 16 Chuvashia 17 Dagestan 18 Evenk 19 Ingush 20 Irkutsk 21 Ivanovo 22 Kabardin-Balkar 23 Kaliningrad 24 Kalmyk 25 Kaluga 26 Kamchatka 27 Karachay-Cherkess 28 Karelia 29 Kemerovo 30 KhabarovskAternity APM Version 11.4.2 253 IP Geography Mappings ScreenCountry Code (Country) Region Code Region31 Khakass 32 Khanty-Mansiy 33 Kirov 34 Komi 36 Koryak 37 Kostroma 38 Krasnodar 39 Krasnoyarsk 40 Kurgan 41 Kursk 42 Leningrad 43 Lipetsk 44 Magadan 45 Mariy-El 46 Mordovia 47 Moskva 48 Moscow City 49 Murmansk 50 Nenets 51 Nizhegorod 52 Novgorod 53 Novosibirsk 54 Omsk 55 Orenburg 56 Orel 57 Penza 58 Perm 59 Primorye 60 Pskov 61 Rostov 62 Ryazan 63 Sakha 64 Sakhalin 65 Samara254 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region66 Saint Petersburg City 67 Saratov 68 North Ossetia 69 Smolensk 70 Stavrop 71 Sverdlovsk 72 Tambovskaya oblast 73 Tatarstan 74 Taymyr 75 Tomsk 76 Tula 77 Tve 78 Tyume 79 Tuva 80 Udmurt 81 Uyanovsk 83 Vladimir 84 Volgograd 85 Vologda 86 Voronezh 87 Yamal-Nenets 88 Yaroslavl 89 Yevrey 90 Permskiy Kray 91 Krasnoyarskiy Kray 92 Kamchatskiy Kray 93 Zabaykalskiy Kray RWA (Rwanda) 1 Butare 6 Gitarama 7Kibungo 9 Kigali 11 Est 12 Kigali 13 NordAternity APM Version 11.4.2 255 IP Geography Mappings ScreenCountry Code (Country) Region Code Region14 Ouest 15 Sud SAU (Saudi Arabia) 2 Al Bahah 5Al Madinah 6 Ash Sharqiyah 8Al Qasim 10 Ar Riyad 11 Asir Province 13 Hail 14 Makkah 15 Al Hudud ash Shamaliyah 16 Najran 17 Jizan 19 Tabuk 20 Al Jawf SRB (Serbia) 1 Kosovo 2Vojvodina SGP (Singapore)SLB (Solomon Islands) 3 Malaita 6 Guadalcanal 7Isabel 8 Makira 9Temotu 10 Central 11 Western 12 Choiseul 13 Rennell and Bellona SYC (Seychelles) 1 Anse aux Pins 2 Anse Boileau 3Anse Etoile 4Anse Louis 5Anse Royale 6 Baie Lazare256 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region7 Baie Sainte Anne 8Beau Vallon 9Bel Air 10 Bel Ombre 11 Cascade 12 Glacis 13 Grand Anse 14 Grand Anse 15 La Digue 16 La Riviere Anglaise 17 Mont Buxton 18 Mont Fleuri 19 Plaisance 20 Pointe La Rue 21 Port Glaud 22 Saint Louis 23 Takamaka SDN (Sudan) 27 Al Wusta 28 Al Istiwa?iyah 29 Al Khartum 30 Ash Shamaliyah 31 Ash Sharqiyah 32 Bahr al Ghazal 33 Darfur 34 Kurdufan 35 Upper Nile 40 Al Wahadah State 44 Central Equatoria State SWE (Sweden) 2 Blekinge Lan 3 Gavleborgs Lan 5Gotlands Lan 6 Hallands Lan 7 Jamtlands Lan 8Jonkopings LanAternity APM Version 11.4.2 257 IP Geography Mappings ScreenCountry Code (Country) Region Code Region9 Kalmar Lan 10 Dalarnas Lan 12 Kronobergs Lan 14 Norrbottens Lan 15 Orebro Lan 16 Ostergotlands Lan 18 Sodermanlands Lan 21 Uppsala Lan 22 Varmlands Lan 23 Vasterbottens Lan 24 Vasternorrlands Lan 25 Vastmanlands Lan 26 Stockholms Lan 27 Skane Lan 28 Vastra Gotaland SHN (St. Helena) 1 Ascension 2 Saint Helena 3 Tristan da Cunha SVN (Slovenia) 1 Ajdovscina Commune 2 Beltinci Commune 3 Bled Commune 4 Bohinj Commune 5 Borovnica Commune 6 Bovec Commune 7 Brda Commune 8 Brezice Commune 9 Brezovica Commune 11 Celje Commune 12 Cerklje na Gorenjskem Commune 13 Cerknica Commune 14 Cerkno Commune 15 Crensovci Commune 16 Crna na Koroskem Commune 17 Crnomelj Commune258 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region19 Divaca Commune 20 Dobrepolje Commune 22 Dol pri Ljubljani Commune 24 Dornava Commune 25 Dravograd Commune 26 Duplek Commune 27 Gorenja vas-Poljane Commune 28 Gorisnica Commune 29 Gornja Radgona Commune 30 Gornji Grad Commune 31 Gornji Petrovci Commune 32 Grosuplje Commune 34 Hrastnik Commune 35 Hrpelje-Kozina Commune 36 Idrija Commune 37 Ig Commune 38 Ilirska Bistrica Commune 39 Ivancna Gorica Commune 40 Izola-Isola Commune 42 Jursinci Commune 44 Kanal Commune 45 Kidricevo Commune 46 Kobarid Commune 47 Kobilje Commune 49 Komen Commune 50 Koper-Capodistria Urban Commune 51 Kozje Commune 52 Kranj Commune 53 Kranjska Gora Commune 54 Krsko Commune 55 Kungota Commune 57 Lasko Commune 61 Ljubljana Urban Commune 62 Ljubno CommuneAternity APM Version 11.4.2 259 IP Geography Mappings ScreenCountry Code (Country) Region Code Region64 Logatec Commune 66 Loski Potok Commune 68 Lukovica Commune 71 Medvode Commune 72 Menges Commune 73 Metlika Commune 74 Mezica Commune 76 Mislinja Commune 77 Moravce Commune 78 Moravske Toplice Commune 79 Mozirje Commune 80 Murska Sobota Urban Commune 81 Muta Commune 82 Naklo Commune 83 Nazarje Commune 84 Nova Gorica Urban Commune 86 Odranci Commune 87 Ormoz Commune 88 Osilnica Commune 89 Pesnica Commune 91 Pivka Commune 92 Podcetrtek Commune 94 Postojna Commune 97 Puconci Commune 98 Race-Fram Commune 99 Radece Commune A1 Radenci Commune A2 Radlje ob Dravi Commune A3 Radovljica Commune A6 Rogasovci Commune A7 Rogaska Slatina Commune A8 Rogatec Commune B1 Semic Commune B2 Sencur Commune260 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionB3 Sentilj Commune B4 Sentjernej Commune B6 Sevnica Commune B7 Sezana Commune B8 Skocjan Commune B9 Skofja Loka Commune C1 Skofljica Commune C2 Slovenj Gradec Urban Commune C4 Slovenske Konjice Commune C5 Smarje pri Jelsah Commune C6 Smartno ob Paki Commune C7 Sostanj Commune C8 Starse Commune C9 Store Commune D1 Sveti Jurij Commune D2 Tolmin Commune D3 Trbovlje Commune D4 Trebnje Commune D5 Trzic Commune D6 Turnisce Commune D7 Velenje Urban Commune D8 Velike Lasce Commune E1 Vipava Commune E2 Vitanje Commune E3 Vodice Commune E5 Vrhnika Commune E6 Vuzenica Commune E7 Zagorje ob Savi Commune E9 Zavrc Commune F1 Zelezniki Commune F2 Ziri Commune F3 Zrece Commune F4 Benedikt Commune F5 Bistrica ob Sotli CommuneAternity APM Version 11.4.2 261 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionF6 Bloke Commune F7 Braslovce Commune F8 Cankova Commune F9 Cerkvenjak Commune G1 Destrnik Commune G2 Dobje Commune G3 Dobrna Commune G4 Dobrova-Horjul-Polhov Gradec Commune G5 Dobrovnik-Dobronak Commune G6 Dolenjske Toplice Commune G7 Domzale Commune G8 Grad Commune G9 Hajdina Commune H1 Hoce-Slivnica Commune H2 Hodos-Hodos Commune H3 Horjul Commune H4 Jesenice Commune H5 Jezersko Commune H6 Kamnik Commune H7 Kocevje Commune H8 Komenda Commune H9 Kostel Commune I1 Krizevci Commune I2 Kuzma Commune I3 Lenart Commune I4 Lendava-Lendva Commune I5 Litija Commune I6 Ljutomer Commune I7 Loska Dolina Commune I8 Lovrenc na Pohorju Commune I9 Luce Commune J1 Majsperk Commune J2 Maribor Commune J3 Markovci Commune262 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionJ4 Miklavz na Dravskem polju Commune J5 Miren-Kostanjevica CommuneJ6 Mirna Pec Commune J7 Novo mesto Urban Commune J8 Oplotnica Commune J9 Piran-Pirano Commune K1 Podlehnik Commune K2 Podvelka Commune K3 Polzela Commune K4 Prebold Commune K5 Preddvor Commune K6 Prevalje Commune K7 Ptuj Urban Commune K8 Ravne na Koroskem Commune K9 Razkrizje Commune L1 Ribnica Commune L2 Ribnica na Pohorju Commune L3 Ruse Commune L4 Salovci Commune L5 Selnica ob Dravi Commune L6 Sempeter-Vrtojba Commune L7 Sentjur pri Celju Commune L8 Slovenska Bistrica Commune L9 Smartno pri Litiji Commune M1 Sodrazica Commune M2 Solcava Commune M3 Sveta Ana Commune M4 Sveti Andraz v Slovenskih goricah Commune M5 Tabor CommuneM6 Tisina Commune M7 Trnovska vas Commune M8 Trzin Commune M9 Velika Polana CommuneAternity APM Version 11.4.2 263 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionN1 Verzej Commune N2 Videm Commune N3 Vojnik Commune N4 Vransko Commune N5 Zalec Commune N6 Zetale Commune N7 Zirovnica Commune N8 Zuzemberk Commune N9 Apace Commune O1 Cirkulane Commune SVK (Slovakia) 1 Banska Bystrica 2 Bratislava 3Kosice 4Nitra 5Presov 6Trencin 7 Trnava 8 Zilina SLE (Sierra Leone) 1 Eastern 2 Northern 3 Southern 4Western Area SMR (San Marino) 1 Acquaviva 2Chiesanuova 3 Domagnano 4Faetano 5Fiorentino 6 Borgo Maggiore 7San Marino 8 Monte Giardino 9 Serravalle SEN (Senegal) 1 Dakar 3Diourbel 5Tambacounda264 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region7Thies 9 Fatick 10 Kaolack 11 Kolda 12 Ziguinchor 13 Louga 14 Saint-Louis 15 Matam SOM (Somalia) 1 Bakool 2 Banaadir 3 Bari 4Bay 5Galguduud 6Gedo 7Hiiraan 8Jubbada Dhexe 9Jubbada Hoose 10 Mudug 11 Nugaal 12 Sanaag 13 Shabeellaha Dhexe 14 Shabeellaha Hoose 16 Woqooyi Galbeed 18 Nugaal 19 Togdheer 20 Woqooyi Galbeed 21 Awdal 22 Sool SUR (Suriname) 10 Brokopondo 11 Commewijne 12 Coronie 13 Marowijne 14 Nickerie 15 ParaAternity APM Version 11.4.2 265 IP Geography Mappings ScreenCountry Code (Country) Region Code Region16 Paramaribo 17 Saramacca 18 Sipaliwini 19 Wanica SDN (South Sudan) 1 Central Equatoria 2 Eastern Equatoria 3Jonglei 4Lakes 5 Northern Bahr el Ghazal 6Unity 7 Upper Nile 8Warrap 9 Western Bahr el Ghazal 10 Western Equatoria STP (Sao Tome and Principe) 1 Principe 2Sao Tome SLV (El Salvador) 1 Ahuachapan 2 Cabanas 3 Chalatenango 4Cuscatlan 5 La Libertad 6 La Paz 7 La Union 8 Morazan 9San Miguel 10 San Salvador 11 Santa Ana 12 San Vicente 13 Sonsonate 14 Usulutan SYR (Syria) 1 Al Hasakah 2 Al Ladhiqiyah 3 Al Qunaytirah 4 Ar Raqqah266 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region5As Suwayd 6Dar 7 Dayr az Zawr 8Rif Dimashq 9 Halab 10 Hamah 11 Hims 12 Idlib 13 Dimashq 14 Tartus SWZ (Swaziland) 1 Hhohho 2Lubombo 3 Manzini 4Shiselweni 5Praslin TCD (Chad) 1 Batha 2Biltine 3 Borkou-Ennedi-Tibesti 4 Chari-Baguirmi 5 Guera 6Kanem 7Lac 8 Logone Occidental 9 Logone Oriental 10 Mayo-Kebbi 11 Moyen-Chari 12 Ouaddai 13 Salamat 14 Tandjile TGO (Togo) 22 Centrale 23 Kara 24 Maritime 25 Plateaux 26 SavanesAternity APM Version 11.4.2 267 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionTHA (Thailand) 1 Mae Hong Son 2Chiang Mai 3Chiang Rai 4Nan 5Lamphun 6Lampang 7Phrae 8Tak 9 Sukhothai 10 Uttaradit 11 Kamphaeng Phet 12 Phitsanulok 13 Phichit 14 Phetchabun 15 Uthai Thani 16 Nakhon Sawan 17 Nong Khai 18 Loei 20 Sakon Nakhon 21 Nakhon Phanom 22 Khon Kaen 23 Kalasin 24 Maha Sarakham 25 Roi Et 26 Chaiyaphum 27 Nakhon Ratchasima 28 Buriram 29 Surin 30 Sisaket 31 Narathiwat 32 Chai Nat 33 Sing Buri 34 Lop Buri 35 Ang Thong268 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region36 Phra Nakhon Si Ayutthaya 37 Saraburi 38 Nonthaburi 39 Pathum Thani 40 Krung Thep 41 Phayao 42 Samut Prakan 43 Nakhon Nayok 44 Chachoengsao 45 Prachin Buri 46 Chon Buri 47 Rayong 48 Chanthaburi 49 Trat 50 Kanchanaburi 51 Suphan Buri 52 Ratchaburi 53 Nakhon Pathom 54 Samut Songkhram 55 Samut Sakhon 56 Phetchaburi 57 Prachuap Khiri Khan 58 Chumphon 59 Ranong 60 Surat Thani 61 Phangnga 62 Phuket 63 Krabi 64 Nakhon Si Thammarat 65 Trang 66 Phatthalung 67 Satun 68 Songkhla 69 PattaniAternity APM Version 11.4.2 269 IP Geography Mappings ScreenCountry Code (Country) Region Code Region70 Yala 71 Ubon Ratchathani 72 Yasothon 73 Nakhon Phanom 74 Prachin Buri 75 Ubon Ratchathani 76 Udon Thani 77 Amnat Charoen 78 Mukdahan 79 Nong Bua Lamphu 80 Sa Kaeo 81 Bueng Kan TJK (Tajikistan) 1 Kuhistoni Badakhshon 2 Khatlon 3Sughd TKM (Turkmenistan) 1 Ahal 2 Balkan 3 Dashoguz 4Lebap 5 Mary TUN (Tunisia) 2 Kasserine 3 Kairouan 6Jendouba 10 Qafsah 14 El Kef 15 Al Mahdia 16 Al Munastir 17 Bajah 18 Bizerte 19 Nabeul 22 Siliana 23 Sousse 27 Ben Arous 28 Madanin270 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region29 Gabes 31 Kebili 32 Sfax 33 Sidi Bou Zid 34 Tataouine 35 Tozeur 36 Tunis 37 Zaghouan 38 Aiana 39 Manouba TON (Tonga) 1 Ha 2Tongatapu 3 Vava TUR (Turkey) 2 Adiyaman 3Afyonkarahisar 4Agri 5Amasya 7Antalya 8Artvin 9Aydin 10 Balikesir 11 Bilecik 12 Bingol 13 Bitlis 14 Bolu 15 Burdur 16 Bursa 17 Canakkale 19 Corum 20 Denizli 21 Diyarbakir 22 Edirne 23 Elazig 24 ErzincanAternity APM Version 11.4.2 271 IP Geography Mappings ScreenCountry Code (Country) Region Code Region25 Erzurum 26 Eskisehir 28 Giresun 31 Hatay 32 Mersin 33 Isparta 34 Istanbul 35 Izmir 37 Kastamonu 38 Kayseri 39 Kirklareli 40 Kirsehir 41 Kocaeli 43 Kutahya 44 Malatya 45 Manisa 46 Kahramanmaras 48 Mugla 49 Mus 50 Nevsehir 52 Ordu 53 Rize 54 Sakarya 55 Samsun 57 Sinop 58 Sivas 59 Tekirdag 60 Tokat 61 Trabzon 62 Tunceli 63 Sanliurfa 64 Usak 65 Van 66 Yozgat272 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region68 Ankara 69 Gumushane 70 Hakkari 71 Konya 72 Mardin 73 Nigde 74 Siirt 75 Aksaray 76 Batman 77 Bayburt 78 Karaman 79 Kirikkale 80 Sirnak 81 Adana 82 Cankiri 83 Gaziantep 84 Kars 85 Zonguldak 86 Ardahan 87 Bartin 88 Igdir 89 Karabuk 90 Kilis 91 Osmaniye 92 Yalova 93 Duzce TTO (Trinidad and Tobago) 1 Arima 2 Caroni 3 Mayaro 4 Nariva 5 Port-of-Spain 6Saint Andrew 7 Saint David 8Saint GeorgeAternity APM Version 11.4.2 273 IP Geography Mappings ScreenCountry Code (Country) Region Code Region9 Saint Patrick 10 San Fernando 11 Tobago 12 Victoria TWN (Taiwan) 1 Fu-chien 2 Kao-hsiung 3Tai-pei 4Tai-wan TZA (Tanzania) 2 Pwani 3 Dodoma 4Iringa 5Kigoma 6 Kilimanjaro 7Lindi 8 Mara 9Mbeya 10 Morogoro 11 Mtwara 12 Mwanza 13 Pemba North 14 Ruvuma 15 Shinyanga 16 Singida 17 Tabora 18 Tanga 19 Kagera 20 Pemba South 21 Zanzibar Central 22 Zanzibar North 23 Dar es Salaam 24 Rukwa 25 Zanzibar Urban 26 Arusha 27 Manyara274 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionUKR (Ukraine) 1 Cherkaska Oblast 2 Chernihivska Oblast 3Chernivetska Oblast 4 Dnipropetrovska Oblast 5 Donetska Oblast 6 Ivano-Frankivska Oblast 7 Kharkivska Oblast 8 Khersonska Oblast 9 Khmelnytska Oblast 10 Kirovohradska Oblast 11 Krym 12 Kyyiv 13 Kyyivska Oblast 14 Luhanska Oblas 15 Lavivska Oblast 16 Mykolayivska Oblast 17 Odeska Oblast 18 Poltavska Oblast 19 Rivnenska Oblast 20 Sevastopol 21 Sumska Oblast 22 Ternopilska Oblast 23 Vinnytska Oblast 24 Volynska Oblast 25 Zakarpatska Oblast 26 Zaporizka Oblast 27 Zhytomyrska Oblast UGA (Uganda) 26 Apac 28 Bundibugyo 29 Bushenyi 30 Gulu 31 Hoima 33 Jinja 36 KalangalaAternity APM Version 11.4.2 275 IP Geography Mappings ScreenCountry Code (Country) Region Code Region37 Kampala 38 Kamuli 39 Kapchorwa 40 Kasese 41 Kibale 42 Kiboga 43 Kisoro 45 Kotido 46 Kumi 47 Lira 50 Masindi 52 Mbarara 56 Mubende 58 Nebbi 59 Ntungamo 60 Pallisa 61 Rakai 65 Adjumani 66 Bugiri 67 Busia 69 Katakwi 70 Luwero 71 Masaka 72 Moyo 73 Nakasongola 74 Sembabule 76 Tororo 77 Arua 78 Iganga 79 Kabarole 80 Kaberamaido 81 Kamwenge 82 Kanungu 83 Kayunga276 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region84 Kitgum 85 Kyenjojo 86 Mayuge 87 Mbale 88 Moroto 89 Mpigi 90 Mukono 91 Nakapiripirit 92 Pader 93 Rukungiri 94 Sironko 95 Soroti 96 Wakiso 97 Yumbe USA (United States) AA Armed Forces Americas AE Armed Forces Europe AK Alaska AL Alabama AP Armed Forces Pacific AR Arkansas AS American Samoa AZ Arizona CA California CO Colorado CT Connecticut DC District of Columbia DE Delaware FL Florida FM Federated States of Micronesia GA Georgia GU Guam HI Hawaii IA Iowa ID IdahoAternity APM Version 11.4.2 277 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionIL Illinois IN Indiana KS Kansas KY Kentucky LA Louisiana MA Massachusetts MD Maryland ME Maine MH Marshall Islands MI Michigan MN Minnesota MO Missouri MP Northern Mariana Islands MS Mississippi MT Montana NC North Carolina ND North Dakota NE Nebraska NH New Hampshire NJ New Jersey NM New Mexico NV Nevada NY New York OH Ohio OK Oklahoma OR Oregon PA Pennsylvania PW Palau RI Rhode Island SC South Carolina SD South Dakota TN Tennessee TX Texas UT Utah278 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code RegionVA Virginia VI Virgin Islands VT Vermont WA Washington WI Wisconsin WV West Virginia WY Wyoming URY (Uruguay) 1 Artigas 2 Canelones 3 Cerro Largo 4 Colonia 5 Durazno 6Flores 7 Florida 8 Lavalleja 9Maldonado 10 Montevideo 11 Paysandu 12 Rio Negro 13 Rivera 14 Rocha 15 Salto 16 San Jose 17 Soriano 18 Tacuarembo 19 Treinta y Tres UZB (Uzbekistan) 1 Andijon 2Bukhoro 3 Farghona 4 Jizzakh 5 Khorazm 6 Namangan 7 Nawoiy 8 QashqadaryoAternity APM Version 11.4.2 279 IP Geography Mappings ScreenCountry Code (Country) Region Code Region9 Qoraqalpoghiston 10 Samarqand 11 Sirdaryo 12 Surkhondaryo 13 Toshkent 14 Toshkent VCT (Saint Vincent and The 1 Charlotte Grenadines)2Saint Andrew 3 Saint David 4Saint George 5 Saint Patrick 6 Grenadines VEN (Venezuela) 1 Amazonas 2 Anzoategui 3Apure 4Aragua 5 Barinas 6Bolivar 7 Carabobo 8Cojedes 9 Delta Amacuro 11 Falcon 12 Guarico 13 Lara 14 Merida 15 Miranda 16 Monagas 17 Nueva Esparta 18 Portuguesa 19 Sucre 20 Tachira 21 Trujillo 22 Yaracuy 23 Zulia280 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region24 Dependencias Federales 25 Distrito Federal 26 Vargas VNM (Vietnam) 1 An Giang 3Ben Tre 5Cao Bang 9Dong Thap 13 Hai Phong 20 Ho Chi Minh 21 Kien Giang 23 Lam Dong 24 Long An 30 Quang Ninh 32 Son La 33 Tay Ninh 34 Thanh Hoa 35 Thai Binh 37 Tien Giang 39 Lang Son 43 Dong Nai 44 Ha Noi 45 Ba Ria-Vung Tau 46 Binh Dinh 47 Binh Thuan 49 Gia Lai 50 Ha Giang 52 Ha Tinh 53 Hoa Binh 54 Khanh Hoa 55 Kon Tum 58 Nghe An 59 Ninh Binh 60 Ninh Thuan 61 Phu YenAternity APM Version 11.4.2 281 IP Geography Mappings ScreenCountry Code (Country) Region Code Region62 Quang Binh 63 Quang Ngai 64 Quang Tri 65 Soc Trang 66 Thua Thien-Hue 67 Tra Vinh 68 Tuyen Quang 69 Vinh Long 70 Yen Bai 71 Bac Giang 72 Bac Kan 73 Bac Lieu 74 Bac Ninh 75 Binh Duong 76 Binh Phuoc 77 Ca Mau 78 Da Nang 79 Hai Duong 80 Ha Nam 81 Hung Yen 82 Nam Dinh 83 Phu Tho 84 Quang Nam 85 Thai Nguyen 86 Vinh Phuc 87 Can Tho 88 Dac Lak 89 Lai Chau 90 Lao Cai 91 Dak Nong 92 Dien Bien 93 Hau Giang VUT (Vanuatu) 5 Ambrym 6Aoba282 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region7Torba 8Efate 9Epi 10 Malakula 11 Paama 12 Pentecote 13 Sanma 14 Shepherd 15 Tafea 16 Malampa 17 Penama 18 Shefa WSM (Samoa) 2 Aiga-i-le-Tai 3Atua 4Fa 5 Gaga 6Va 7 Gagaifomauga 8Palauli 9 Satupa 10 Tuamasaga 11 Vaisigano YEM (Yemen) 1 Abyan 2Adan 3Al Mahrah 4Hadramawt 5Shabwah 6Lahij 7 Al Bayda 8 Al Hudaydah 9Al Jawf 10 Al Mahwit 11 Dhamar 12 HajjahAternity APM Version 11.4.2 283 IP Geography Mappings ScreenCountry Code (Country) Region Code Region13 Ibb 14 Mas rib 15 Sas dah 16 San 17 Taizz 18 Ad Dali 19 Amran 20 Al Bayda’s 21 Al Jawf 22 Hajjah 23 Ibb 24 Lahij 25 Taizz ZAF (South Africa) 1 North-Western Province 2 KwaZulu-Natal 3 Free State 5Eastern Cape 6Gauteng 7Mpumalanga 8 Northern Cape 9 Limpopo 10 North-West 11 Western Cape ZMB (Zambia) 1 Western 2Central 3Eastern 4Luapula 5 Northern 6 North-Western 7 Southern 8 Copperbelt 9 Lusaka ZWE (Zimbabwe) 1 Manicaland 2 Midlands284 Aternity APM Version 11.4.2 IP Geography Mappings ScreenCountry Code (Country) Region Code Region3 Mashonaland Central 4 Mashonaland East 5 Mashonaland West 6 Matabeleland North 7 Matabeleland South 8 Masvingo 9 Bulawayo 10 HarareAternity APM Version 11.4.2 285 IP Geography Mappings Screen286 Aternity APM Version 11.4.2 CHAPTER 21 Search Context Configuration ScreenNote: This Screen Not Available to Read-Only Users Configuration screens are available only when you log in to APM using an administrative account. See the “User Roles“ topic for details on administrative and read-only accounts. This screen specifies search strings users can choose in the “Search Area“ of the “Search Tab“ to further limit searches. Users choose from search context strings in a list that appears in the “Show the top“area. For example:If there are no search context strings defined in this tab, the For list does not appear in the Find the Top area. A typical use for search contexts is to create search strings that differentiate among different applications that APM is monitoring. APM implicitly adds the search string using the AND logical operator. The string is not visible in the search text, but is added implicitly as follows: After any user-supplied search string Before any “Search Analysis Operators“Search Context Dialog Use the Search Context dialog to create or modify search context strings. Click Add a Search Context to create a new search context string. Click the edit icon ( ) in the row for an existing search context string to modify that string. Supply these values in the dialog box that opens: Display Name: the name that will appear in the search context listAternity APM Version 11.4.2 287 Search Context Configuration Screen Description: an optional description that appears when users select the search context and pause the mouse pointer over the name Query: the search string to implicitly add to the user’s search. The search string can specify “Compound Filter Conditions“, but cannot specify an analysis operator. Click Save to close the dialog box and save changes.Search Context String Examples Use or adapt the following examples to get started with search context strings:Display Name QueryBack End Only server.responsetime > 0 and ! ( pageload.responsetime >= 0 ) Front End Only !(server.responsetime >=0) and pageload.responsetime >=0 Front End with Back End server.responsetime >=0 and pageload.responsetime >=0 Has SQL sql=* Has URL url=* Has URL Outbound urloutbound=* 288 Aternity APM Version 11.4.2 CHAPTER 22 Custom Reports ScreenNote: This Screen Not Available to Read-Only Users Configuration screens are available only when you log in to APM using an administrative account. See the “User Roles“ topic for details on administrative and read-only accounts. Reports combine output from multiple “Search Analysis Operators“ and display it on a single page. Use this screen to define your own reports that present output from useful searches and analysis operators. Users run reports with the “report“ analysis operator in the “Search Tab“. Use the value of the Display Name column as the value of the -name argument of the report analysis operator. For example: | report -name example reportThis screen lists any custom report definitions that have been created.  To create a report definition, click Define a Custom Report. In the dialog box that opens, supply a custom report definition in the text box. To get started, adapt the example report as described in “Quick Start“.  To edit an existing report, click the edit icon ( ) and make changes to the report definition.  Click the run report icon ( ) next to the report name to open the “Search Tab“ and run the report. Quick StartReports are a collection of one or more analysis operators that users run at the same time with the “report“ analysis operator in the “Search Tab“. Recall that analysis operators accept the matching transactions from any search, perform additional processing, and summarize the data in graphs and tables. Aternity APM Version 11.4.2 289 Custom Reports ScreenThe first step in creating any report is to develop a search and accompanying analysis operator in the Search tab. For example, suppose you want a report to contain this search and analysis operator result:Once you have your search and analysis operator working the way you want, follow these steps to create a report:1) Copy the search and analysis operator string: urlpath = *trade* | transactioncount -group_by urlpath -visualization barchart -limit 10 -format no_timelegend -time_rollup 1 week 2) Create a report definition that includes the string (shown here in bold) in the “search“ report item. For example: version,1 name,My First Report title,My really-easy first report description,A example to show how easy creating custom reports can be! header,Top URLs text,These are the top Trade URLs based on transaction count. Usually you'd add an explanation here to help the reader interpret this section. search,urlpath = *trade* | transactioncount -group_by urlpath -visualization barchart -limit 10 -format no_timelegend -time_rollup 1 week 3) In the Custom Reports screen, click Define a Custom Report. The Custom Reports dialog opens. Paste the report definition in the text box and click Save.290 Aternity APM Version 11.4.2 Custom Reports Screen4) Click the run-report icon ( ) next to My First Report. This will open the Search tab and run the report.Using Report Items and Variables This section describes how to use report items to structure a report and introduces variables. See “Report Items Reference“ for complete details. This section uses the following example report definition: Example 1 name,example report title,Example Report description,This is an example report version,1 variable,SERVER,my_database_server header,First section header2,Let's make some charts search, server=$$SERVER$$ | transactioncount -group_by url header2,Let's search by server search, server=$$SERVER$$ | transactioncount -group_by server header,Another section search, server=$$SERVER$$ | barchart text, Now let's add some related searchesAternity APM Version 11.4.2 291 Custom Reports Screen relatedsearch,View calls data for top $$TOPN$$ transactions, | calls relatedsearch,Transactioncount by Server, | transactioncount -group_by server relatedsearch,Transactioncount by User, | transactioncount -group_by userThe following diagram shows how report items in the example report definition correspond to its output in the Search tab:Follow these steps to create the report definition and customize its output:1) Copy the report from Example 1.2) Click Define a Custom Report. The Define a Custom Report dialog opens. Paste the example report in the text box and click Save.292 Aternity APM Version 11.4.2 Custom Reports Screen3) The new report, example report, appears in the report list. Click the run-report icon ( ) next to the name to open the Search tab and run the report. The report includes a table of contents that correspond to “header1“ and “header2“ report items. Sections that follow contain details for each header. Notice that the searches did not work and returned empty tables and graphs. This is because the search report items specify the $$SERVER$$ variable reference. The report definition assigns the value my_database_server to the SERVER variable, which is not likely to match a server name in your environment. 4) Click the edit icon ( ) and edit the report definition to fix the problem. Find the variable report item and replace my_database_server with the name of a server in your environment. In this example, the server name is nhx1-w2k8r2-12: variable,SERVER,nhx1-w2k8r2-12Aternity APM Version 11.4.2 293 Custom Reports Screen5) Run the report again using a valid value for the SERVER variable. Now the output contains the expected tables and charts for the “transactioncount“ analysis operator:Troubleshooting If the report is missing required items (“version“ and “name“), clicking Save in the Custom Report dialog box fails. The dialog box closes but an error appears at the bottom of the browser window:Run a custom report by clicking the run report icon ( ) next to the report name. If you do not see the all the report elements you expect, check the report items for misspellings. Misspelling the item name (for example, specifying searh instead of search) will cause it not to appear in the report.294 Aternity APM Version 11.4.2 Custom Reports ScreenIf there is a error in the syntax for a search string or analysis operator, a red error message appears at the top of the report and in the section for the corresponding operator:As described in “Report Items Reference“, the “search“ item has restrictions that result in errors if they are not observed. The entire report fails with a red error message. For example, if you omit an analysis operator in the search item, the report fails with Unable to determine operator name: For example:Report Items ReferenceThe following usage notes apply to all report items: Report items must start on a new line.  Precede comments with the number sign (#). Any line starting with # is ignored. Use lowercase letters for report item names (header1 will work; HEADER1 will not). Except for “name“, “title“, “description“, “version“, and “variable“, the position of report items in the definition controls their placement in the corresponding output. Aternity APM Version 11.4.2 295 Custom Reports ScreenUsing Double Quotation Marks to Escape Special Characters Enclose any string that contains commas in double quotation marks: text,"If you want a comma, enclose the argument in quotes"If the string also contains double quotation marks, enclose the entire string in double quotation marks and use another double quotation mark as an escape character. For example, consider this search string that uses an “extract expression“ with both double and single quotation marks: (exceptioncount > 0) | transactioncount -group_by "extract(exception, '^([^: ]+)([: ]|$)' , 1 )" -limit 8To use this search string in a report definition, precede each double quotation mark with another: search,"exceptioncount > 0 | transactioncount -group_by ""extract(exception, '^([^: ]+)([: ]|$)' , 1 )"" -limit 8"If the string begins with a double quotation mark, enclose the string in double quotation marks and escape the double quotation marks with another double quotation mark. This results in a string that begins with three double quotation marks. For example, to display this search string in a report: "Find the top" value is: $$TOPN$$Add this item to the report definition: text,"""Find the top"" value is: $$TOPN$$" versionThis report item is required and specifies the version number of the report framework: version,1 nameThis report item is required. It specifies the text that appears in the auto complete text in the Search tab search area. (The text also appears in the Display Name column in the list of reports in the Custom Reports screen.) Refresh the browser to see new report names in the search auto-complete. If there are multiple name items in the same definition, APM uses the value of the first one. For example, this name item: name,this appears as the auto-complete value in the search text box Results in: titleThis report item specifies the text that appears at the top of the report. If there are multiple title items in the same definition, APM uses the value of the first one.296 Aternity APM Version 11.4.2 Custom Reports ScreenFor example, this title item: title,This Value Appears At The Top Of The Report Results in: descriptionThis report item specifies text that appears immediately below the title text, if it is present, and above the table of contents. If there are multiple description items in the same definition, APM uses the value of the first one. For example, this description item: description, use the description to summarize the report Results in:To control where explanatory text appears in the report, use “text“ report items. variableThis report item specifies a variable name and assigned value using this form: variable,variablename,variablevalue Note the following:  You must declare the variable before referring to it.  Report items that follow the declaration can refer to the variable using the form $$variablename$$. APM replaces variable references with the value.  Variable names are case sensitive (var1 is not the same as VAR1). You can refer to another previously-declared variable within a variable declaration. The “name“ report item does not support variable references.  TOPN and TOPN_PHRASE are built-in variables that you do not declare and should not override. They are automatically assigned values related to the “Find the top” setting in the Search screen. (This setting specifies the maximum number of transactions a search returns.) –TOPN is replaced by the numeric value following the “Find the top” label in the Search tab. –TOPN_PHRASE is replaced by a phrase based on that numeric value and the value of the limit criterion following that value (“Slowest Transactions”, “Fastest Transactions”, or “Most Exceptions”).Aternity APM Version 11.4.2 297 Custom Reports ScreenRefer to the built-in variables like other variables. For example: text, Value of TOPN built-in variable: $$TOPN$$ text, Value of TOPN_PHRASE built-in variable: '$$TOPN_PHRASE$$' These references return text similar to the following: Value of TOPN built-in variable: 10000 Value of TOPN_PHRASE built-in variable: 'top 10,000 slowest matching transactions' The following variable item specifies a system name that a subsequent search item refers to: variable,SERVER,nhv1-w2k12-2 search, server=$$SERVER$$ | transactioncount -group_by url When the report runs, APM replaces the references with the value:The following example shows how you can nest variable references: name,variables usage title, Using Variables version,1 variable, VAR1, this is VAR1 variable, VAR2, VAR2 refers to VAR1: $$VAR1$$ variable, VAR3, VAR3 refers to VAR2: $$VAR2$$ text, $$VAR1$$ text, $$VAR2$$ text, $$VAR3$$Running this report results in: header1 header2The header1 and header2 report items specify headings that define report sections and appear as links in the table of contents.298 Aternity APM Version 11.4.2 Custom Reports ScreenHeadings have a navigation button that appears when you pause the mouse pointer over the heading text. Click the button to move to the top of the report. For example, this report definition: name,headers usage version,1 header1, Heading 1 header2, Heading 2 header1, Heading 1 header2, Heading 2Results in: searchThis report item specifies a search string that invokes an analysis operator. It can specify any valid search string (including no search string) as long as it invokes an analysis operator. The following example does not specify a search string and invokes the “statistics“ analysis operator: header1, Heading 1 search, | statistics -compare_days_ago 1Aternity APM Version 11.4.2 299 Custom Reports ScreenThis report definition generates the following output:Note that the report includes the search string as a link before the output of the analysis operator. Click the link to open a new browser tab or window that runs the search for the same time range. Keep in mind the following restrictions: The search string must invoke an analysis operator. For example, this search argument does not: name, search without an operator search, apptype = 'java' Running the report (| report -name search without an operator) generates an error similar to the following: Unable to determine operator name from:'apptype = 'java''. Check your report definition in search without an operator  A search item cannot invoke other reports. If it does, the report runs but includes a recursion error. For example, this report definition invokes itself: name, recursive report search, | report -name example_report Invoking the report ( | report -name recursive report) generates this error: Recursion limit reached: This operator cannot launch another operator Similarly, the search string cannot invoke the navigator analysis operator. If it does, the report runs but includes the same error as the previous example. This is because the navigator operator invokes timeseries, which exceeds the recursion limit.  A report cannot contain multiple search arguments that invoke the histogram analysis operator. For example, this report definition invokes histogram twice: name,2 histogram report version,1 search, | histogram search, | histogram300 Aternity APM Version 11.4.2 Custom Reports ScreenInvoking the report fails with the following error: Only one histogram operator is allowed to be used in a report at a time. See report definition for '2 histogram report' in 2histogram_report.csv textThis report item specifies explanatory text that appears where you specify it (as compared to the “description“ item, which appears above the table of contents). This report definition shows text items in different locations in a report: name,text example version,1 description, Description text appears above the table of contents text, Text before heading 1 header1, Heading 1 text, Text after heading search, | rivergraph text, Text after search header2, Heading 2 text, Text after heading 2Running the report displays this result:Aternity APM Version 11.4.2 301 Custom Reports Screen htmlThis report item specifies an HTML snippet to add to the report. This is useful to add custom images and links. The following example adds a company logo and a link to this documentation: name,html examples version,1 html, html, <A HREF="/help/wwhelp/wwhimpl/js/html/wwhelp.htm?context=analysis&topic=search" target="docWindow">Search Tab help topic</A>Running the report displays the logo and link: relatedsearchThis report item specifies a link that opens a new browser tab or window with a related search using this form: relatedsearch,linktext,searchstring Unlike the “search“ item, the search string for relatedsearch does not have to invoke an analysis operator. It can specify any valid APM search. Specifying relatedsearch creates a Related Search subheading. Each relatedsearch link is rendered as a bulleted item following the subheading. For example, this report definition: name,relatedsearch example header1, Heading 1 search, | statistics -compare_days_ago 1 relatedsearch,Related searches don't have to invoke an analysis operator, apptype=java relatedsearch,Response Time by Transaction Type, | rivergraph -metric duration -exclude_weekends -group_by transactiontype302 Aternity APM Version 11.4.2 Custom Reports ScreenResults in:Aternity APM Version 11.4.2 303 Custom Reports Screen304 Aternity APM Version 11.4.2 CHAPTER 23 Docker Container Display Name ScreenNote: This Screen Not Available to Read-Only Users Configuration screens are available only when you log in to APM using an administrative account. See the “User Roles“ topic for details on administrative and read-only accounts. This screen specifies options that control how Docker container names will appear in the APM interface. (See “Monitoring Docker Containers“ for details on configuring the agent to monitor Docker environments.) The options display different combinations of “Docker Container Tags“ values: Container Name shows the value of the “container name“ tag Container Name and Host shows the value of the “container name“ and “docker hostname“ tags Container Id and Host shows the value of the “container id“ and “docker hostname“ tags Container Image Name and Id shows the value of the “container image“ and “container id“ tagsAternity APM Version 11.4.2 305 Docker Container Display Name Screen306 Aternity APM Version 11.4.2 CHAPTER 24 Import/Export/Reset Settings ScreenNote: This Screen Not Available to Read-Only Users Configuration screens are available only when you log in to APM using an administrative account. See the “User Roles“ topic for details on administrative and read-only accounts. This screen manages analysis server settings: “Import and Export Configuration“ applies or writes many of the settings made through CONFIGURE menu options. “Reset User Settings to Default“ undoes changes to many display settings in “Data Analysis Tabs and Metric Data Views“.Import and Export Configuration Use “Import Configuration“ to apply various configuration settings from another analysis server or restore an earlier exported backup. Use “Export Configuration“ to write settings for use in another analysis server or to back up current configuration settings Exporting and importing analysis server configuration settings can make deploying analysis servers faster and easier: In environments with many analysis servers that you want to configure the same way, you can configure one analysis server, export its settings, and quickly import the settings to the other analysis servers. Aternity APM Version 11.4.2 307 Import/Export/Reset Settings Screen If you need to remove and reinstall the analysis server, you can restore settings.Note: Importing Configuration Settings Overwrites Current Settings When you import configuration settings, the imported settings overwrite any existing settings. Note: Do Not Modify Exported Settings Do not modify the file generated by clicking Generate Configuration File in “Export Configuration“. Changing settings by editing the file is not supported.What Settings Are Exported Not all configuration settings are exported. The following table summarizes which settings are and are not exported. Exported NOT Exported Related Help Topic NotesProcess instrumentation “Define a Configuration configurations Screen“ Transaction types “Transaction Types“ Use the Export/Import feature in the Manage Transaction Types screen to modify multiple transaction types. See “Bulk Editing Transaction Types with Export and Import“. Application definitions “Define an Application Use the Export/Import Screen“ feature in the Manage Applications screen to modify multiple application definitions. See “Bulk Editing Application Definitions with Export and Import“. Custom mappings of IP “IP Geography to geographic location Mappings Screen“ Alert definitions “Alert Definitions Screen“ Environmental and “Threshold Overview transactional thresholds Screen“ Filter rules “Filter Rules Screen“ Custom category colors “Customize Category Color Screen“ Custom reports for the “Custom Reports Search tab Screen“308 Aternity APM Version 11.4.2 Import/Export/Reset Settings ScreenExported NOT Exported Related Help Topic NotesSearch contexts “Search Context Configuration Screen“Alert email templates “Alert Notification Template Configuration Screen“ Licenses “License Configuration Screen“ Agent details (including “Agent Details Screen“ process names) User accounts and roles “SteelCental SaaS - Account“Import ConfigurationNote: Importing Configuration Settings Overwrites Current Settings When you import configuration settings, the imported settings overwrite any existing settings. Click Upload Configuration File to open a file browser and navigate to a previously downloaded configuration settings file. Open the file to import those configuration settings. Select Generate backup of current configuration to export the current configuration settings before starting the import operation. This creates a backup that appears with any other in the Configuration Files list in the “Export Configuration“ area. You can effectively roll back the import operation by importing the backup. This is useful if you decide you do not want to use the imported configuration settings. (Note that the import operation automatically rolls back to the current settings if there is an error, even if Generate Backup setting is not selected. In this case, an error (Could not import data) appears and the original settings remain.) Export ConfigurationClick Generate Configuration File to write configuration settings for use in another analysis server or to back up current configuration settings. A link to the generated settings file appears in the Configuration Files list. The Configuration Files list shows up to five of the most recently generated configuration files. Click a link to copy the file to your local system.Reset User Settings to Default Click Reset... to undo changes to many display settings in “Data Analysis Tabs and Metric Data Views“, including the following: Edit settings in cards (see “Card Reference“ for details)  Edit settings in the “Application Map Tab“ Aternity APM Version 11.4.2 309 Import/Export/Reset Settings Screen Sort order and column selections in the “Search Results“ table in the “Search Tab“. Having Dismissed the “Welcome” 60-second tour 310 Aternity APM Version 11.4.2 CHAPTER 25 Diagnostic Bundle ScreenNote: This Screen Not Available to Read-Only Users Configuration screens are available only when you log in to APM using an administrative account. See the “User Roles“ topic for details on administrative and read-only accounts. A “diagnostic bundle” is a compressed package of analysis server log files, configuration settings, and output from commands and troubleshooting scripts. Typically, you generate a diagnostic bundle at the request of Aternity support. Use this screen to:  Download diagnostic bundles for agents (created from the “Agent Details Screen“). Enable “Diagnostic Mode“. Download Diagnostic Bundles This area shows links to download previously created agent diagnostic bundles. Only the two most-recently generated agent diagnostic bundles appear here, and only if they were created in the last day. Older diagnostic bundles are deleted as newer bundles are created. Agent Diagnostic Bundles This area shows links to copy any previously-created diagnostic bundles for agents. (You generate diagnostic bundles for agent by clicking the “Download the agent logs and diagnostic files“ link in the “Agent Details Screen“.) Click a link to copy the file to your local system. Diagnostic ModeSelect this option to show advanced details, such as additional fields in search results, throughout AppInternals. Enable diagnostic mode to help troubleshooting problems in AppInternals.Aternity APM Version 11.4.2 311 Diagnostic Bundle ScreenThe effects of enabling diagnostic mode include: In the “Data Analysis Tabs and Metric Data Views“, the ability to refresh cards by clicking their title. This is useful for troubleshooting because it causes the query that the card makes to be written to one of the log files (silo_query-queries.log) included in analysis server diagnostic bundle.  In the “Search Tab“, additional transaction search fields in auto-complete suggestions and in the Add Criteria option’s drop-down list. In the “Transaction Details“ window: – Display of additional fields in the “Call Details“ section – Display of additional data in the “Resources Tab“ section  In the “System Status Screen“, display of the Advanced System Status Information section.312 Aternity APM Version 11.4.2 CHAPTER 26 Filter Rules ScreenNote: This Screen Not Available to Read-Only Users Configuration screens are available only when you log in to APM using an administrative account. See the “User Roles“ topic for details on administrative and read-only accounts. This screen displays JSON text that specifies filters that prevent uninteresting transactions from being indexed and stored in the APM database. Use this screen to modify the default filters. This is not required and is typically not necessary. As described in “Filter Field Values“, filters can specify criteria including the category, class, and method of the top-level call in the transaction. Such filters match ONLY the top-level “root” call.OverviewSome transactions are not useful in diagnosing performance issues. Most applications have internal processing that may generate slow transactions but that are not performance problems. For example, Java application servers may have internal threads that routinely make long-running socket calls as part of normal processing.Aternity APM Version 11.4.2 313 Filter Rules ScreenBecause they are long running, transactions from these calls can dominate search results in the “Search Tab“. The following example shows how internal socket calls appear as the slowest over the time period:Also, because such calls are often the only call in the thread, their transactions are not interesting for users to investigate further in the “Transaction Details“ window. To address these issues, APM by default excludes transactions that are not useful for diagnosing performance issues. This filtering reduces uninteresting “noise” transactions without affecting interesting transactions. You can change this behavior by modifying the default filters. This is not required and is typically not necessary.Default FiltersUse the default filters as examples you can adapt. For example, the following filters exclude transactions containing a single top-level Java socket call to specific methods: { "rootcomponent": "remote", "rootmethodname": "read", "rootclassname": "java.net.SocketInputStream", "maxcallcount": 1 }, { "rootcomponent": "remote", "rootmethodname": "write", "rootclassname": "java.net.SocketOutputStream",314 Aternity APM Version 11.4.2 Filter Rules Screen"maxcallcount": 1 },Some of the default filters are less restrictive. For example, these filters exclude transactions containing top-level calls to any methods in specific Java socket classes: { "rootclassname": "java.net.SocksSocketImpl" }, { "rootclassname": "java.net.AbstractPlainSocketImpl" },Modifying the Default FiltersYou can modify the existing filters or add new ones. Adapt the default filters to suit your purposes. Note: Be Careful—Filters Can Exclude Transactions Required to Connect Related Transactions Exercise care in creating or modifying filters. It is possible to exclude interesting transactions. In particular, filters can exclude transactions that APM requires for detecting cross-tier activity. When this happens, it effectively disables the ability to include related transactions in the “Transaction Details“ window. Make changes to the JSON text in the text box and click Save. If you supply invalid JSON, the screen displays an error and disables the Save button. (You can use an external JSON and validator, such as http://jsonlint.com/, for more-detailed validation.Filter Usage NotesEach filter specifies one or more field-value pairs: Most fields correspond to search fields you can specify in the Search tab. No wildcards are allowed. Each value must specify a complete match for the corresponding field.  Values are not case sensitive. Only those transactions that match all the specifications of the filter will not be indexed or stored in the database. In other words, APM uses a logical AND operation on filter criteria to determine whether a transaction matches and will be excluded.Filter Field ValuesThe following table lists valid fields in a filter specification. Links in the table are to descriptions in the “Transaction Search Reference“ material:Field Name Description instance The application “instance“ for the transaction. maxcallcount The maximum number of calls in the transaction. Transactions with a larger number of calls will not be excluded. Typically, specify a value of 1 in combination with other fields to avoid filtering out interesting transactions.Aternity APM Version 11.4.2 315 Filter Rules ScreenField Name Description maxduration The maximum “responsetime“ of the transaction. Transactions with a longer duration will not be excluded. minduration The minimum “responsetime“ of the transaction. Transactions with a shorter duration will not be excluded. This is useful for methods that can be long-running in some cases and short-lived in others. With this field, you can exclude the long-running cases. rootcomponent The “category“ of the top-level “root” call in the transaction. Use the “rootcomponent“ search field to directly search for these values when creating filter rules. rootclassname The “class“ of the top-level “root” call in the transaction. Use the “rootclass“ search field to directly search for these values when creating filter rules. rootmethodname The “method“ of the top-level “root” call in the transaction. Use the “rootmethod“ search field to directly search for these values when creating filter rules.Filters are evaluated only against transactions that APM stores after you make changes. They have no effect on earlier transactions.Restoring Default FiltersThis section shows the complete set of default filters. You can use this to restore the default behavior if you want to revert from changes that did not behave as expected. Copy and paste this text in its entirety to the text box and click Save. { "filters": [ { "rootmethodname": "read", "rootclassname": "java.net.SocketInputStream", "maxcallcount": "1" }, { "rootmethodname": "write", "rootclassname": "java.net.SocketOutputStream", "maxcallcount": "1" }, { "rootmethodname": "connect", "rootclassname": "System.Net.Sockets.Socket", "maxcallcount": "1" }, { "rootmethodname": "close", "rootclassname": "System.Net.Sockets.Socket", "maxcallcount": "1" }, { "rootmethodname": "send", "rootclassname": "System.Net.Sockets.Socket", "maxcallcount": "1" }, { "rootmethodname": "EndReceive", "rootclassname": "System.Net.Sockets.Socket", "maxcallcount": "1" },316 Aternity APM Version 11.4.2 Filter Rules Screen{ "rootmethodname": "accept", "rootclassname": "org.apache.jk.common.ChannelSocket", "maxcallcount": "1" }, { "rootmethodname": "accept", "rootclassname": "System.Net.Sockets.Socket", "maxcallcount": "1" }, { "rootclassname": "java.net.SocksSocketImpl" }, { "rootclassname": "java.net.AbstractPlainSocketImpl" }, { "rootclassname": "com.sun.net.ssl.internal.ssl.AppInputStream" }, { "rootclassname": "com.sun.net.ssl.internal.ssl.AppOutputStream" }, { "rootclassname": "com.ibm.jsse.a" }, { "rootclassname": "com.ibm.jsse.b" }, { "rootclassname": "com.ibm.jsse2.e" }, { "rootclassname": "com.ibm.jsse2.j" }, { "rootclassname": "sun.security.ssl.AppOutputStream" }, { "rootclassname": "sun.security.ssl.AppInputStream" }, { "rootclassname": "java.net.PlainSocketImpl" } ], "xthread_suppress_agent_version_regex": [ "10\\.15\\.1\\..*" ] }Aternity APM Version 11.4.2 317 Filter Rules Screen318 Aternity APM Version 11.4.2 CHAPTER 27 License Configuration ScreenNote: This Screen Not Available to Read-Only Users Configuration screens are available only when you log in to APM using an administrative account. See the “User Roles“ topic for details on administrative and read-only accounts. Use this screen to see detailed APM license status (the “License Status“ area) and to add new licenses (the “Set License“ area). APM licenses have an expiration date and a fixed number of “license units”: After the expiration date, the AppInternals interface will not display data when users log in. Agents continue to collect data and the analysis server continues to harvest it. Monitored processes running on agent systems consume license units. Processes potentially use a license unit if the “Harvest Option“ setting is enabled (in the “Edit Processes to Instrument Dialog“ of the “Agent Details Screen“). See “License Unit Usage Rules“ for details on license units. The analysis server itself is not licensed. When APM is first installed, it operates under a trial license. Trial licenses provide 10 license units for up to 14 days after the first user logs in to AppInternals. After 14 days, the trial license expires and the AppInternals interface will not display data after users log in. License Unit Usage RulesAPM licenses contain a fixed number of license units. Processes use license units as follows: Every Java process (JVM) uses a license unit.  Every agent with one or more .NET processes uses a single license unit. Multiple .NET processes on the same agent consume only one unit. Licensing affects only whether the analysis server retrieves transaction trace data from agents. It does not affect page-view data enabled through the “Collect End-User Experience Data“ option in the “Define a Configuration Screen“. The analysis server itself is not licensed. Aternity APM Version 11.4.2 319 License Configuration ScreenThe following figure shows different combinations of Java and .NET processes running on an agent host and how many license units they use:You can see how many license units are in use in different places in the APM interface: The “License Status“ area of this screen summarizes license unit use across all agents. The “Agent List Screen“ shows how many licenses units each agent uses when you sort by licenses. The Agent Details area of the “Agent Details Screen“ shows how many license units a particular agent uses. The “Edit Processes to Instrument Dialog“ dialog in the “Agent Details Screen“ shows the number of remaining license units overall.Managing License Unit UsageThis section describes options that control whether and how agents use license units.Control Usage at Agent Level: the Release Licenses on Disconnect OptionThe Release Licenses on Disconnect option in the “Actions“ list of the “Agent Details Screen“ causes the analysis server to release any license units used by an agent when it cannot connect to that agent (in other words, when the agent status is “Offline“). This is typically useful in cloud environments, where agent virtual machines can be short-lived and re-initialize frequently as a different virtual machine under a different host name. However, you can set any agent to release licenses when the analysis server cannot connect to it, which offers additional flexibility to manage license unit usage. Control Usage at Process Level: the Harvest Option Note: Any process that uses a license unit will continue to use one even if it goes down. Use the Harvest option as described here to release a license unit for a process that is consuming one.320 Aternity APM Version 11.4.2 License Configuration ScreenUse the Harvest option to control whether processes use a license unit. This option controls whether the analysis server retrieves transaction traces for specific processes and debits license units accordingly. The Harvest option is separate from the Instrument option. If you disable harvesting for a process but leave instrumentation enabled, the process will continue to generate transaction trace data. However, instrumentation must be enabled before you can enable or disable harvesting for a process. Enable or disable the Harvest option in any of the following screens:  The “Agent Details Screen“: – For all the processes on the agent, with the “Start Harvesting All“ action – For a single process, in the “Edit Processes to Instrument Dialog“ (the Harvest option is dimmed and unavailable if enabling would consume a license unit and none is available) The “Process List Screen“, for all selected processes, with the Start Harvesting action The analysis server monitors the number of license units in use. If enabling harvesting for processes would use more than the available number of license units, an error message appears at the bottom of the screen:In cases where there are not enough license units, some operations in the Bulk Process Configuration screen may succeed partially. For example, if you only have one remaining license unit, but select five processes (that will use a license unit) and click Start Harvesting, harvesting is enabled only on a single process. Release a license unit by clearing the Harvest setting for a process that is using one. One way to do this is in the Bulk Process Configuration screen:1) Sort the list of processes by clicking the Harvest column so that processes with Harvest enabled are at the top of the list.2) Select ALL of the .NET processes on an agent OR any Java process. Aternity APM Version 11.4.2 321 License Configuration Screen3) Click Actions, then Stop Harvesting.License StatusThis area shows details about the current license and how many license units are in use.The details under the Usage graph show the distribution of consumed license units across JVMs and hosts that with at least one .NET process. The numbers are links that open the “Process List Screen“ filtered to show the corresponding processes to instrument: x JVMs filters the processes to show those JVMs with collection enabled. 322 Aternity APM Version 11.4.2 License Configuration Screen y .NET Hosts filters the processes to show all .NET processes on hosts with at least one process with collection enabled.Set LicenseAfter the trial license expires, you must buy an APM license and obtain an activation key. Click the link below (or in the license screen) to start the purchase process: http://www.riverbed.com/appinternals/buy/?s_rbp=product The Set License area shows the basic the steps to add a new license, described in more detail here. After you buy a license, you will receive an e-mail message from Aternity support. The e-mail provides credentials to log on to the Aternity support site. You use the Activation Request String from this screen to obtain the license key and activate the license. The activation request string uniquely identifies the analysis server system. Follow these steps:1) Log in to the APM interface and navigate to the Configure > Licensing screen. Click the icon to copy the Activation Request String. You use it in step 5).2) Log in to the Aternity support site using the credentials in the e-mail message.3) Navigate to Licenses > OPNET Software Licenses. 4) Enter your Group ID and Serial Number from the e-mail message and click Search. Find the serial number in the search results. Click allocate nodes to a console:Aternity APM Version 11.4.2 323 License Configuration Screen5) The Enter Activation Request String window opens. In the Enter Activation Request String window, paste the Activation Request String from this screen and click Submit:6) The Add Nodes to a Console window opens. In the Add Nodes to a Console window: – Supply the fully-qualified analysis server host name as the Console Name. – Specify the number of agents you want to license with this analysis server in the Add Nodes to this console field. – Click Confirm. This opens the Activation Key window and adds a row for the host name to the licenses list.324 Aternity APM Version 11.4.2 License Configuration Screen7) In the Activation Key window, copy the activation key. Paste it in the Paste the 'Activation Key' area in this screen: 8) Click Save.9) Log out and back in to the APM interface to activate the license. APM checks license status only when users log in, so all users must log out and in to take advantage of a new license.License Status Messages and MeaningAPM checks license status only when users log in. Here are possible license status messages:Message NotesWarning: Your free trial Trial licenses provide full functionality up to 14 days after the first user logs in to license will soon expire APM. With a trial license, this message appears 10 minutes after users log in. After on <date>. 14 days, the trial license expires and the APM interface will not display data. Warning: this license will When a valid (non-trial) license is within 30 days of expiring this message appears soon expire on <date>. 10 minutes after users log in. Warning: Your license has This message appears after a license expires. Users can access all screens in the APM expired. interface, but see “no data” messages or errors where data should display. In addition, License is invalid messages appear on some screens. Unable to determine If APM cannot determine the license state, it displays this message and the behavior license status. is the same as with an expired license. Exceeding license count An operation (such as enabling the Harvest option for a process) used more license limit of <n> units than are available. See “Managing License Unit Usage“ for steps to release a license unit.Aternity APM Version 11.4.2 325 License Configuration ScreenMessage NotesInvalid 'Activation Key’. The value in the Paste the 'Activation Key' area is invalid. Possible causes are: Please verify that the • The activation key value copied from Aternity support site was somehow entire key was correctly corrupted (for examples, characters were removed or changed). copied & pasted and is for this instance of • The activation key does not correspond to the analysis server system (as AppInternals. identified by the Activation Request String in the “Set License“area). This can happen if the activation request string was incorrectly pasted in the Aternity support site to generate the activation key. Error saving license The analysis server service that processes licenses is not running or not responding. 'Activation Key'.326 Aternity APM Version 11.4.2 CHAPTER 28 Customize Category Color ScreenNote: This Screen Not Available to Read-Only Users Configuration screens are available only when you log in to APM using an administrative account. See the “User Roles“ topic for details on administrative and read-only accounts. Use this screen to customize the colors that represent categories in charts in the analysis server interface. See the “Categories Reference“ topic for descriptions of the different categories. To change the color for a category:1) Click the edit icon ( ) in the row for the category whose color you want to change. 2) The Edit Color dialog box opens.3) In the Custom Color setting, supply the hexadecimal color code for the new color. You must precede the color code with #. You can use web sites such as http://htmlcolorcodes.com/color-picker/ to find the color code for different colors. 4) Click Save. To remove all custom colors that have may have been specified, click Restore Default. Aternity APM Version 11.4.2 327 Customize Category Color Screen328 Aternity APM Version 11.4.2 CHAPTER 29 System Status ScreenNote: This Screen Not Available to Read-Only Users Configuration screens are available only when you log in to APM using an administrative account. See the “User Roles“ topic for details on administrative and read-only accounts. The System Status Information screen shows details about the analysis server system. Note that the values on this page do not update automatically. You must refresh the web page display to see changes. (Or you can navigate to another screen and back.) Section Data DisplayedSystem Health Overall status and resource consumption on the analysis server system. Click the Services link to expand the list and see the status of all analysis server services. (Note that the analysis server runs multiple copies of some services, so you will see multiple entries with the same service name.) CPU utilization, Disk Busy, and Network Rates are averages over the last 5 minutes (samples are taken every 15 seconds). The other values are a snapshot at the time the screen opened or was last refreshed. Storage • Details of disk usage on the analysis server. When disk usage exceeds 90 percent of the available disk space, APM deletes the oldest data it has stored. • In a cluster environment, disk utilization across each node in the cluster is displayed. Incoming Data Details on the volume of application trace data, environmental metric data, and EUE data (end-user-experience generated by browsers on monitored web pages). Advanced System This area of the screen appears only when the “Diagnostic Mode“ option in the Status Information “Diagnostic Bundle Screen“ screen is set. The internal processing detail it shows can be useful for Aternity support. Aternity APM Version 11.4.2 329 System Status Screen330 Aternity APM Version 11.4.2 CHAPTER 30 SAML Configuration This screen opens from the SAML Configuration option in the “SteelCental SaaS - Account“ screen. It is available only when this feature is enabled for the company (also called realm) by Aternity APM SaaS operations. This screen provides settings to configure authentication using a SAML (Security Assertion Markup Language) server. Using a SAML server is useful to take advantage of single sign-on, where users do not have to supply credentials after they have authenticated with your organization’s “identity provider” (IdP) server. For an introduction to SAML, see the Wikipedia article Security Assertion Markup Language. Behavior with SAML Single Sign-On In SAML terms, the application that uses the authentication server (the AppInternals analysis server) is a “service provider” (SP), the user’s browser is a “user agent”, and the authentication server is the “identity provider” (IdP). With SAML single sign-on, users whose APM user account was created with a User Type of SAML (see “Adding Users: SAML-Enabled Companies“) log in just once. At the sign-in screen, they provide the canonical email address for their account and click Sign In. The browser session is redirected to the IdP server. If the user does not already have a valid session, the IdP server prompts for credentials and validates the user. If the user has a valid session, the IdP server does not prompt for credentials. In either case, the IdP server redirects the browser session to the analysis server as a signed-in user. Once validated via an IdP server, users are not prompted to log in again when they use another company or product (such as SteelCentral Aternity) that uses the same authentication server. Aternity APM Version 11.4.2 331 SAML ConfigurationThe following diagram shows the log-in sequence (credit: CC BY-SA 3.0, Wikipedia)Prerequisites Before you configure the analysis server to use an IdP server, confirm that: The IdP server supports the SAML 2.0 protocol.  The IdP server sends the user's full “canonical” email address to the analysis server as the user identifier. You must supply the same canonical email address in the analysis server Add User screen (see “Adding Users: SAML-Enabled Companies“ in the “SteelCental SaaS - Account“ topic). 332 Aternity APM Version 11.4.2 SAML ConfigurationConfiguring SAML authenticationFollow these steps to configure SAML authentication:1) Contact your IT department to exchange details for interacting with the IdP server used by your organization. Provide the SP Entity ID and SP Consumer URL values shown on this screen to the IdP server: – SP Entity ID: The URL of the analysis server login page for single sign-on – SP Consumer URL: The URL to which the user is redirected after successful authentication through the IdP server. 2) Select the Enable SAML Based Authentication checkbox.3) The IdP server provides XML metadata that contains configuration and integration details for single sign-on. Obtain this metadata and copy it in the IdP Metadata area.4) Select the Sign Authentication Request checkbox if the IdP expects that authentication requests it receives from the analysis server will be signed with security certificates. This setting should correspond to the value of the WantAuthnRequestsSigned attribute in the IdP Metadata area. If the value of WantAuthnRequestsSigned is 0 or false, do not select this field. Similarly, if you are using Active Directory Federation Services (ADFS) as your IdP, do not select this field.Aternity APM Version 11.4.2 333 SAML Configuration334 Aternity APM Version 11.4.2 CHAPTER 31 Environmental Thresholds, The Wildcard Character, And TagsOverviewBeginning with 10.21.0, APM supports using the wildcard character (*) and tags when defining “Environmental Thresholds“, which is particularly useful for cloud-native environments where server/container names are unique/ephemeral.Note: You create user-defined tags in the “Agent Details Screen“.Because servers are ephemeral in a containerized environment, defining thresholds against a single server is ineffective, since the server may cease to exist before a violation is triggered. To ensure that thresholds work as expected in a containerized environment, AppInternals supports tags in environmental threshold definitions, since tags represent entities (and groups of entities) that come and go. The wildcard character (*) is also supported on both tags and server names, to extend environmental threshold definitions across a group of tags or servers that match a specified pattern.Creating Environmental Thresholds With Tags And The Wildcard CharacterTo create an environmental threshold using tags and the wildcard character, follow these steps:1. Log in to the Analysis Server WebUI as admin.2. In the menu bar, navigate to Configure->Thresholds.The Threshold Overview screen appears. Aternity APM Version 11.4.2 335 Environmental Thresholds, The Wildcard Character, And Tags3. From the Threshold Overview screen, select Add an Environmental Threshold, and the Add Environment Threshold window appears, as follows:4. From the Add Environment Threshold window, you can select a server and/or a tag, and both fields accept the wildcard character. Note that at least one field must be populated.• The following screen illustrates defining an environmental threshold using the server field with a wildcard, so that any server that matches the pattern will trigger a threshold violation:336 Aternity APM Version 11.4.2 Environmental Thresholds, The Wildcard Character, And Tags• The following screen illustrates defining an environmental threshold using a tag, so that any server that matches the tag will trigger a threshold violation. Note that this field also accepts the wildcard character.• The following screen illustrates defining an environmental threshold using both a server and a tag with a wildcard character in the server field, so that any server that matches the pattern or the specified tag will trigger a violation. Note that both fields accept the wildcard character.5. Once you have created and saved the Environment Thresholds, you can select them when defining alerts in the “Define an Alert Screen“ in Config->Alert Definitions->Define an Alert.Aternity APM Version 11.4.2 337 Environmental Thresholds, The Wildcard Character, And Tags338 Aternity APM Version 11.4.2 Agent System Configuration and MetricsThe following chapters explains the APM configuration tasks you can perform after the agent is installed: “Changing the Agent’s Analysis Server“ “Proxy Server Configuration“ “Controlling Agent Software on Windows Systems“ “Controlling Agent Software on Unix-Like Systems“ “Automatic Process Instrumentation on Windows“ “Instrumentation Techniques and Troubleshooting“ “Deploying Agents in Dynamic Environments“ “Deploying the Agent as an NPM Data Source“  “Environmental Metrics and Sub-Agents Reference“ Additionally, the following chapters in the agent installation section explain configuration tasks that are done during the agent installation: “Synchronizing Agent Time With the Analysis Server“  “Enabling Instrumentation of Java Processes“ “Enabling Instrumentation of .NET Processes“ See the “Web Interface Configuration“ topics for details on configuration screens in the analysis server web interface. Aternity APM Version 11.4.2 1 2 Aternity APM Version11.4.2 CHAPTER 1 Synchronizing Agent Time APM requires that times on agent systems and the analysis server be synchronized through a network time service (such as Network Time Protocol (NTP) or Windows Time Service). Before the agent begins generating performance data, make sure time is synchronized as described in the “Synchronizing Agent Time With the Analysis Server“ installation topic.Aternity APM Version 11.4.2 3 Synchronizing Agent Time4 Aternity APM Version11.4.2 CHAPTER 2 Changing the Agent’s Analysis Server This section describes how to change the analysis server that an agent connects to when it starts. The agent installation prompts for details to connect to the analysis server. Typically, you do not have change this configuration after installation but can do so as described here. Follow these steps:1) Stop the agent. – On Windows systems, stop the Riverbed SteelCentral AppInternals Agent Controller service. See “Controlling Agent Software on Windows Systems“ for details. – On UNIX systems, use the agent stop command. See “Controlling Agent Software on Unix-Like Systems“ for details. 2) Edit the file <installdir>\Panorama\hedzup\mn\data\dsa.xml and change the value of attributes. Which attributes you change depends on whether the agent will connect to an analysis server installed in your environment (“on premises”) or to the Software as a Service (SaaS) offering of the analysis server (SteelCentral SaaS). – For an on-premises analysis server, change the AnalysisServerHost attribute. If necessary, set the SaaSAnalysisServerEnabled attribute to false. For example: <Attribute name="AnalysisServerHost" value="myAnalysisServer.example.com"/> <Attribute name="SaaSAnalysisServerEnabled" value="false"/> – For the SaaS analysis server, supply the CustomerID for your analysis server. The customer ID is on the“Install Agents Screen“ of the SaaS analysis server. If necessary, set the SaaSAnalysisServerEnabled attribute to true. For example: <Attribute name="CustomerID" value="cd23f7f7-5b82-4a60-b6f0-a4108c59800f"/> <Attribute name="SaaSAnalysisServerEnabled" value="true"/>3) Restart the agent.Aternity APM Version 11.4.2 5 Changing the Agent’s Analysis Server6 Aternity APM Version11.4.2 CHAPTER 3 Proxy Server ConfigurationThe agent installation asks whether the agent will use a proxy server to connect to the analysis server and, if so, details about the proxy server. This section describes changing these settings after installing the agent: Edit the dsa.xml file to enable using a proxy server and specify the host and port. Run the proxy_cred_util script to specify or change credentials for a proxy server.Enabling and Setting the Proxy Server in dsa.xml To enable and set a proxy server for the agent, edit the file <installdir>\Panorama\hedzup\mn\data\dsa.xml. Follow these steps:1) Stop the dsa process on the agent. – On Windows systems, stop the Riverbed SteelCentral AppInternals Agent Controller service or use the agent stop dsa command. See “Controlling Agent Software on Windows Systems“ for details. – On UNIX systems, use the agent stop dsa command. See “Controlling Agent Software on Unix-Like Systems“ for details. 2) Edit dsa.xml and locate the ProxyServer attributes. If the initial installation did not specify any proxy server, the attributes appear as follows: <Attribute name="ProxyServerEnabled" value="FALSE"/> <Attribute name="ProxyServerHost" value=""/> <Attribute name="ProxyServerPort" value=""/>3) Change the value of the attributes to enable the server and specify a host and port. For ProxyServerHost, supply a system name, fully-qualified domain name (FQDN), or IP address. For example: <Attribute name="ProxyServerEnabled" value="TRUE"/> <Attribute name="ProxyServerHost" value="nhx1-w2k8r2-19"/> <Attribute name="ProxyServerPort" value="4080"/>4) If necessary, supply authentication credentials as described in “Changing Credentials with the proxy_cred_util Script“.5) Restart the agent.Aternity APM Version 11.4.2 7 Proxy Server ConfigurationChanging Credentials with the proxy_cred_util Script If the proxy server requires authentication, use the proxy_cred_util.sh script in the <installdir>/Panorama/hedzup/mn/support directory (<installdir>\Panorama\hedzup\mn\support\proxy_cred_util.bat on Windows systems) to supply or change credentials. The agent supports proxy servers that use Basic and Digest authentication. It does not support NTLM authentication. Use the -u argument to specify a user name and password separated by a colon. Use the -r argument to specify a realm, if necessary. For example: C:\Panorama\hedzup\mn\support>proxy_cred_util.bat -u proxy1:password 2017-07-28 16:27:03.076:INFO::main: Logging initialized @143ms to org.eclipse.jetty.util.log.StdE rrLog proxy1 OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v null Verification of proxy settings has not been requested.The credentials (with the password obfuscated) are stored in the file <installdir>\Panorama\hedzup\mn\. C:\Panorama\hedzup\mn\support>type ..\security\.proxy.auth #Arguments used for basic and digest proxy authentication #Fri Jul 28 16:27:03 EDT 2017 user=proxy1 password=OBF\:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v Confirming Successful Connection to the Proxy Server Check the DSA.txt log file in the <installdir>\Panorama\hedzup\mn\log\ directory to confirm that the agent connected to the proxy server. Messages similar to the following show that the agent connected successfully: 07/28/2017 03:39:19 PM, dsa , 0x000022cc , INFO , Using proxy server ho st: nhx1-w2k8r2-19 port: 4080 07/28/2017 03:39:29 PM, dsa , 0x00002a24 , INFO , Websocket: Proxy serv er is enabled 07/28/2017 03:39:29 PM, dsa , 0x00002a24 , INFO , Websocket: Proxy serv er host: nhx1-w2k8r2-19 port: 4080If the host is incorrect: 07/28/2017 03:55:15 PM, dsa , 0x000041cc , WARN , Websocket: Unable to connect to Analysis Server at S_10.46.35.80_443_/wsconnect due to UnknownHostException. Reason: B OGUSnhx1-w2k8r2-19If the port is incorrect: 07/28/2017 04:02:59 PM, dsa , 0x00002608 , WARN , Websocket: Unable to connect to Analysis Server at S_10.46.35.80_443_/wsconnect due to ConnectException. Reason: Conne ction refused: no further informationIf the credentials are incorrect: 07/28/2017 04:16:57 PM, dsa , 0x00003c18 , WARN , Websocket: Unable to connect to Analysis Server at S_10.46.35.80_443_/wsconnect due to HttpResponseException. Reason: Unexpected HttpContentResponse[HTTP/1.1 407 Proxy Authentication Required - 2020 bytes] for HttpR equest[CONNECT 10.46.35.80:443 HTTP/1.1]@92c9ffb8 Aternity APM Version11.4.2 Proxy Server ConfigurationAternity APM Version 11.4.2 9 Proxy Server Configuration10 Aternity APM Version11.4.2 CHAPTER 4 Controlling Agent Software on Windows Systems This section describes controlling agent software on Windows systems  Using the Windows service created by the agent installation  From the command line, using the agent.bat file Starting and Stopping the Agent as a Windows ServiceThe APM installation creates a Windows service called Riverbed SteelCentral AppInternals Agent Controller. This service starts and stops the agent (the Dsaserver.exe process), sub-agents, and related processes on Windows systems. The controller service starts or stops all agent processes in one step, which is typically what you want to do. As installed, the service starts automatically when the agent system reboots.  Use the Windows Services utility to start, stop, and check the status of the controller service. Invoke the Services utility from the Administrative Tools folder of the Windows Control Panel: Use the agent.bat file with the start or stop arguments: C:\Panorama\hedzup\mn\bin>agent start Starting the Agent... [started] PID: 10628 Name: dsa Command: [.\DsaServer.exe, -d] [started] PID: 14460 Name: agentrt Command: [.\agentrt_agent.exe, -c] [started] PID: 13284 Name: dotnetagent Command: [.\dotnetagent2.exe, -d] [stopped] Name: npm Command: [.\npm_agent.exe] [stopped] Name: osda Command: [os_agent.exe, -c]Aternity APM Version 11.4.2 11 Controlling Agent Software on Windows Systems Use the net start or net stop command and specify the AgentController service name from a command prompt window: C:\Panorama\hedzup\mn\bin>net start agentcontroller The Riverbed SteelCentral AppInternals Agent Controller service is starting. The Riverbed SteelCentral AppInternals Agent Controller service was started successfully. Tools such as Process Explorer show the processes that the agent controller starts: Starting and Stopping Agent Processes with agent.bat You can also start agent processes using the agent.bat command file in the <installdir>\Panorama\hedzup\mn\bin\ directory. The agent.bat command file uses the same syntax as the agent script on Unix-like operating systems. See “Controlling Agent Software on Unix-Like Systems“ for details. If no additional arguments are specified, the start, stop, and restart arguments control all sub-agents and the DSA. This is the same as starting or stopping the Riverbed SteelCentral AppInternals Agent Controller service. You can also specify an additional argument to control an individual agent process. This example shows the status of agent processes, then stops and starts the osda sub-agent: C:\Panorama\hedzup\mn\bin>agent.bat status [started] PID: 10172 Name: dsa Command: [.\DsaServer.exe, -d] [started] PID: 14732 Name: agentrt Command: [.\agentrt_agent.exe, -c] [started] PID: 12412 Name: dotnetagent Command: [.\dotnetagent2.exe, -d] [started] PID: 9508 Name: npm Command: [.\npm_agent.exe] [started] PID: 15332 Name: osda Command: [.\os_agent.exe, -c]C:\Panorama\hedzup\mn\bin>agent.bat stop osda Stopping process osda... Stopped osda [started] PID: 10172 Name: dsa Command: [.\DsaServer.exe, -d] [started] PID: 14732 Name: agentrt Command: [.\agentrt_agent.exe, -c] [started] PID: 12412 Name: dotnetagent Command: [.\dotnetagent2.exe, -d] [started] PID: 9508 Name: npm Command: [.\npm_agent.exe] [user stopping] PID: 15332 Name: osda Command: [.\os_agent.exe, -c]C:\Panorama\hedzup\mn\bin>agent.bat status [started] PID: 10172 Name: dsa Command: [.\DsaServer.exe, -d] [started] PID: 14732 Name: agentrt Command: [.\agentrt_agent.exe, -c] [started] PID: 12412 Name: dotnetagent Command: [.\dotnetagent2.exe, -d] [started] PID: 9508 Name: npm Command: [.\npm_agent.exe] [stopped] Name: osda Command: [.\os_agent.exe, -c]C:\Panorama\hedzup\mn\bin>agent.bat start osda Starting osda... Started osda [started] PID: 10172 Name: dsa Command: [.\DsaServer.exe, -d]12 Aternity APM Version11.4.2 Controlling Agent Software on Windows Systems[started] PID: 14732 Name: agentrt Command: [.\agentrt_agent.exe, -c] [started] PID: 12412 Name: dotnetagent Command: [.\dotnetagent2.exe, -d] [started] PID: 9508 Name: npm Command: [.\npm_agent.exe] [started] PID: 9620 Name: osda Command: [.\os_agent.exe, -c]C:\Panorama\hedzup\mn\bin>agent.bat status [started] PID: 10172 Name: dsa Command: [.\DsaServer.exe, -d] [started] PID: 14732 Name: agentrt Command: [.\agentrt_agent.exe, -c] [started] PID: 12412 Name: dotnetagent Command: [.\dotnetagent2.exe, -d] [started] PID: 9508 Name: npm Command: [.\npm_agent.exe] [started] PID: 9620 Name: osda Command: [.\os_agent.exe, -c]Aternity APM Version 11.4.2 13 Controlling Agent Software on Windows Systems14 Aternity APM Version11.4.2 CHAPTER 5 Controlling Agent Software on Unix-Like SystemsThe agent script starts and stops the agent (the dsa process) and sub-agents on Unix-like operating systems. You must start the agent and sub-agents after installation and upgrades using the agent script. Otherwise, you typically only restart the agent during troubleshooting or if it stops responding. (You do not have to restart agent software after rebooting. The installation creates a script and link that stops and starts agent software automatically when the system shuts down and reboots.) The following table shows the agent commands to start the agent and sub-agents, the associated process names, and links to details on the metrics that the sub-agents report. Component agent Command to Start Process(es) Details Agent ./agent start dsa dsa OS sub-agent ./agent start os os_agent “OS Sub-Agent: Unix-Like Operating Systems“ JIDA sub-agent none: starts with Java java “JIDA Sub-Agent“ application JMX sub-agent ./agent start agentrt java “JMX Sub-Agent“ NPM sub-agent ./agent start npm npm_agent “NPM Sub-Agent“ npm_worker Processes sub-agent ./agent start agentrt agentrt_agent “Processes Sub-Agent“The agent script resides in the <installdir>/Panorama/hedzup/mn/bin directory. Run the script from that directory or add the directory to the search path environment variable. The script requires a command argument (start, stop, restart, or status) and, optionally, arguments that specify the target component. For example: ./agent start dsa ./agent start osFor all but the status command, you must be logged in to the account specified during the installation to use agent. If not, agent generates an error: # ./agent stop agent: ERROR You must be running as 'admin' to execute this script.Aternity APM Version 11.4.2 15 Controlling Agent Software on Unix-Like SystemsSyntax agent {start|stop|restart|status} [dsa|agentrt|npm|osda] Arguments start Starts the DSA or the specified sub-agent, or all sub-agents and the DSA, if no additional arguments are specified: bash-4.1$ ./agent start Starting the Agent... [started] PID: 3953 Name: dsa Command: [./dsa, -d] [started] PID: 3958 Name: agentrt Command: [./agentrt] [started] PID: 3971 Name: npm Command: [./npm_agent] [started] PID: 3980 Name: osda Command: [./os_agent] stop Stops the DSA or the specified sub-agent, or all sub-agents and the DSA, if no additional arguments are specified: bash-4.1$ ./agent stop dsa Stopping process dsa... Stopped dsa [user stopping] PID: 2666 Name: dsa Command: [./dsa, -d] [started] PID: 2675 Name: agentrt Command: [./agentrt] [started] PID: 2695 Name: npm Command: [./npm_agent] [started] PID: 2706 Name: osda Command: [./os_agent] restart Stops and starts the DSA or the specified sub-agent, or all sub-agents and the DSA, if no additional arguments are specified: bash-4.1$ ./agent restart Restarting Agent... Stopping process osda Stopping process npm Stopping process agentrt Stopping process dsa Restarting the Agent... [started] PID: 4265 Name: dsa Command: [./dsa, -d] [started] PID: 4271 Name: agentrt Command: [./agentrt] [started] PID: 4282 Name: npm Command: [./npm_agent] [started] PID: 4287 Name: osda Command: [./os_agent] status Displays the process status of the DSA and sub-agents: bash-4.1$ ./agent status [started] PID: 4265 Name: dsa Command: [./dsa, -d] [started] PID: 4271 Name: agentrt Command: [./agentrt] [started] PID: 4282 Name: npm Command: [./npm_agent] [started] PID: 4287 Name: osda Command: [./os_agent]16 Aternity APM Version11.4.2 CHAPTER 6 Automatic Process Instrumentation on Windows This chapter describes the Riverbed Process Injection Driver (RPID) that enables instrumentation for Java and .NET processes on Windows systems. It also includes “rpictrl.exe Command Reference“, a reference to commands for managing RPID. Using RPID is recommended to automatically enable instrumentation and simplify configuration on Windows systems. See “Enabling Instrumentation of Java Processes“ for alternative approaches to using RPID for Java processes on Windows systems.Starting and Enabling RPIDAs described in “Information Required for Installation“ in the installation documentation, the agent installation on Windows systems the Enable Instrumentation Automatically option to start RPID. If you do not choose this option, start RPID using rpictrl.exe enable (this has the same effect as choosing the installation option): C:\Users\Administrator>cd "\Program Files (x86)\Riverbed\ProcessInjection\bin"C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe enable Succesfully enabled process injection.C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe statusRiverbed Process Injection Control Driver Version 10.13.529.0 Copyright 2014-2017. Riverbed Technology. All rights reserved.Start type: system start Status: runningSee “rpictrl.exe Command Reference“ for details on all rpictrl.exe commands.How RPID WorksRPID is a Windows driver (rpid.sys) that watches for processes that start and the modules they load. The agent installation creates RPID and related files in the %ProgramFiles(x86)%\Riverbed\ProcessInjection\bin\ directory.Aternity APM Version 11.4.2 17 Automatic Process Instrumentation on WindowsWhen RPID detects a new .NET or Java process, it injects the appropriate Riverbed process injection library (RPIL, distinct from the RPID driver): rpilj64.dll, rpilj.dll, rpilc.dll, or rpilc64.dll. The injection library checks whether the process is configured for instrumentation (as shown in the analysis server “Agent Details Screen“). If the process is configured for instrumentation, the injection library loads the profiler library that instruments the process and generates performance data. RPID also performs access checks on the RPIL DLLs. If the user account of the target application does not have permissions to execute the RPIL DLLs, RPID allows the application to run, but not instrumented. In such cases, RPID logs a warning message to the system event log indicating the user account and the RPIL DLL that was attempted to be used. Advantages of Automatic Instrumentation RPID simplifies configuration for instrumentation and provides other advantages: RPID automatically provides injection into all .NET and Java processes without the use of the COR_* environment variables, JAVA_TOOL_OPTIONS, command line, script modifications, environment variables, or any other user intervention. There is no need to set environment variables, change command lines, or use tools such as JidaRegister.exe to set JAVA_TOOL_OPTIONS. When environment variables are used for Java or .NET (JAVA_TOOL_OPTIONS or COR_*) to load the profiler library, the environment variables are not always picked up by newly created processes. This occurs on Windows because any new child process typically inherits the environment of its parent process. Unless the parent process restarts, it may not pick up the changes to the environment variables. When using RPID, this is not an issue as RPID does not require the use of environment variables. For .NET applications, RPID allows overriding a different profiler that is still installed (an already-configured COR_PROFILER value). .NET allows for only one profiler to be used within a process. RPID will override other configured profilers on the system. Without RPID, Java applications will not start after removing agent software, unless you also remove JAVA_TOOL_OPTIONS or -agentpath references to RPILJ, the APM Java profiler library. Because it does not rely on specifying file paths, RPID does not have this requirement.  Java applications do not have to specify the 64-bit or 32-bit version of the profiler library to match the architecture of the application. This is often an issue when using JAVA_TOOL_OPTIONS or the –agentpath command line option.  RPID allows instrumentation of applications with an “embedded JVM” that does not use the Java command line launcher. Processes that do not use the Java launcher may not have a mechanism for setting JVM options such as –agentpath. Without using RPID, it can be difficult to instrument native applications that host or embed a JVM. Tomcat.exe is an example of an application with an embedded JVM. rpictrl.exe Command Reference Control process injection on Windows systems with the rpictrl.exe utility. This section describes rpictrl.exe commands. Run the utility from the Windows command prompt in the "%ProgramFiles(x86)%\Riverbed\ProcessInjection\bin\" directory. 18 Aternity APM Version11.4.2 Automatic Process Instrumentation on WindowsYou must run rpictrl.exe as administrator. Even if you are logged in to the Administrator account, depending on User Account Control (UAC) settings, you may have to right-click and choose Run as administrator when you start the command prompt. If you do not have sufficient privileges, rpictrl.exe generates an error: c:\>"%ProgramFiles(x86)%\Riverbed\ProcessInjection\bin\rpictrl.exe" stats Error contacting RPID. Access is denied. helpShow the current version of RPID and valid rpictrl.exe commands. statusDisplay the version of the driver, start type, current status, and if any of the safety features have been activated and blocked the driver from executing. The status command is a useful first command to use when managing RPID. C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe statusRiverbed Process Injection Control Driver Version 10.10.586.0 Copyright 2014-2017. Riverbed Technology. All rights reserved.Start type: system start Status: runningC:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exeThis example shows the status after the driver was deactivated by USD: C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe statusRiverbed Process Injection Control Driver Version 0.0.0.0 Copyright 2014-2017. Riverbed Technology. All rights reserved.Start type: system start Status: stopped USD Deactivated: usd:info can be used for more infoSee “usd“ for details on USD. startStart the driver. Use the start command if the driver was previously stopped with the “stop“ command. If the “status“ command shows a start_type of disabled, the start command will not work. Use the “enable“ command to both enable and start the driver. C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe start Succesfully started process injection. C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe statusRiverbed Process Injection Control Driver Version 0.0.0.0 Copyright 2014-2017. Riverbed Technology. All rights reserved.Start type: system start Status: runningAternity APM Version 11.4.2 19 Automatic Process Instrumentation on WindowsNote that the start command also fails after the driver was deactivated by USD: C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe start Error starting process injection. This driver has been blocked from loadingSee “usd“ for details on USD. stopStop the driver. The stop command stops the driver only until the next reboot. After the system next reboots, the driver will again be started. Use the “disable“ command to stop the driver even after a reboot. C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe stop Successfully stopped process injection.C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe statusRiverbed Process Injection Control Driver Version 0.0.0.0 Copyright 2014-2017. Riverbed Technology. All rights reserved.Start type: system start Status: stopped enableEnable the driver for use and start it. C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe statusRiverbed Process Injection Control Driver Version 0.0.0.0 Copyright 2014-2017. Riverbed Technology. All rights reserved.Start type: disabled Status: stoppedC:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe enable Succesfully enabled process injection.C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe statusRiverbed Process Injection Control Driver Version 0.0.0.0 Copyright 2014-2017. Riverbed Technology. All rights reserved.Start type: system start Status: running disableStop the driver, if it running, and disable it so it will not automatically start after system reboots. C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe statusRiverbed Process Injection Control Driver Version 10.10.586.0 Copyright 2014-2017. Riverbed Technology. All rights reserved.Start type: system start Status: running20 Aternity APM Version11.4.2 Automatic Process Instrumentation on WindowsC:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe disable Successfully disabled process injection.C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe statusRiverbed Process Injection Control Driver Version 10.10.586.0 Copyright 2014-2017. Riverbed Technology. All rights reserved.Start type: disabled Status: stopped statsDisplay various statistics about RPID activitiy since it was started. The statistics are useful in determining if RPID is monitoring the system for injection purposes. The stats command displays the following information:Statistic MeaningStart time Time when RPID was last started.Access check Number of access check violations that have occurred since RPID was started. violationsTotal .NET (32-bit) Total number of .NET 32-bit injections that have occurred since RPID was started. This does not mean that all of those processes are currently running. This is just a running total of all of the injections that have occurred. Total .NET (64-bit) Total number of .NET 64-bit injections that have occurred since RPID was started. This does not mean that all of those processes are currently running. This is just a running total of all of the injections that have occurred. Total Java (32-bit) Total number of Java 32-bit injections that have occurred since RPID was started. This does not mean that all of those processes are currently running. This is just a running total of all of the injections that have occurred. Total Java (64-bit) Total number of Java 64-bit injections that have occurred since RPID was started. This does not mean that all of those processes are currently running. This is just a running total of all of the injections that have occurred. Monitoring Count of the number of processes currently being monitored by RPID. These processes may or may not require injection. RPID attempts to monitor all of the processes on the system and when the need for injection occurs within a process, it then performs injection. This count is simply the number of currently running processes that RPID is aware of on the system. For example: C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe statsStart time: Wednesday, May 31, 2017 5:34:04 PM Access check violations: 0 Total .NET (32-bit): 14 Total .NET (64-bit): 199 Total Java (32-bit): 0 Total Java (64-bit): 1 Monitoring: 6Aternity APM Version 11.4.2 21 Automatic Process Instrumentation on Windows vercheckCheck if the current Windows operating system version is supported by RPID. Use the vercheck command if RPID fails to start to rule out an unsupported Windows version. For example, if RPID was enabled on a supported Windows version and then upgraded to an unsupported version, use vercheck to see if the current Windows version is supported. This example shows that RPID supports the Windows version (Window 2008 R2): C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe vercheck OS Version: 6.1 (Server) Supported: YesThis example shows that RPID does NOT support the Windows version (Window 7 workstation): C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe vercheck OS Version: 6.1 (Workstation) Supported: No Unsupported Reason: Workstation listList currently-running processes that RPID detected and details on their injection status. The list command shows injection-related information for processes on the system. There are two forms: list displays all of the currently-running processes that have some injection characteristic as detected by RPID. This is useful for focusing on processes related to injection. list:all displays all currently-running processes on the system and their injection status. This is useful for ruling out that a process is associated with injection. For example:The list command displays a question mark (?) when it is unable to retrieve the information for that field. For each process, both forms of the command show the following details that can be used for troubleshooting injection issues:Field DescriptionImage name The name of the application or image associated with the processPID The identifier of the process22 Aternity APM Version11.4.2 Automatic Process Instrumentation on WindowsField DescriptionProcessor Architecture The processor architecture, 64 (64-bit) or 32 (32-bit) Session ID The session identifier associated with the process. Session zero is used for servicesInjection Attributes A series of flags that identify various injection attributes of the process. Each attribute type is shown in pairs of Java (j) and .NET (n) with the exception of the last flag (r) that indicates if the process was injected by RPID. From left to right, the attributes indicate: • If the process is Java or .NET capable. Basically this means that the process has loaded a library which is capable of creating a JVM or CLR. • Whether the Riverbed process injection library (RPIL) has been injected into the process. • Whether a JVM or CLR has been created while RPIL has been loaded. Note that RPIL has to be loaded for this flag to accurately reflect the existence of a JVM or CLR • Whether the AppInternals profiler library is currently loaded into the process • Whether the Aternity profiler library is currently loaded into the process • Whether RPID performed injection: r indicates that RPID performed the injection, as opposed to another method such as JAVA_TOOL_OPTIONS or COR_* environment variables or the –agentpath Java command line option.The following figure shows the correspondence of flags to the injection attributes. pid <pid> Display injection details for a specified process. Use the “list“ command to find a process of interest and its process identifier (PID). Then use the pid command to see more details on that process: Full path to the main module of the process. Product name as found in the main module Product version as found in the main moduleAternity APM Version 11.4.2 23 Automatic Process Instrumentation on Windows The injection information for each type of injection that has occurred within the specified process. The type of injection can be one of or a combination of Java and .NET depending upon the process’ use of frameworks. For example, the following list command shows a single instrumented Java process (because of the presence of a j in the seventh flag position), Tomcat7.exe: C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe list controller.exe 1304 64 0 j-j-j-----r SMSvcHost.exe 1752 64 0 -n-n-n----r XenGuestAgent.Exe 2328 64 0 -n-n-n----r WmiPrvSE.exe 2788 64 0 -n-n------r SMSvcHost.exe 3008 64 0 -n-n-n----r XenDPriv.exe 4384 64 0 -n-n-n----r msdtc.exe 5012 64 0 -n-n-n----r Tomcat7.exe 4900 32 0 j-j-j-j---r procexp64.exe 4968 64 2 -n-n------rUse the pid command to see more injection details about that process: C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe pid 4900C:\Tomcat 7.0\bin\Tomcat7.exe Commons Daemon Service Runner 1.0.15.0RPIL Java (32) ------Time: Friday, June 02, 2017 11:56:05 AM Event level: Information Event: RPIL loadedTime: Friday, June 02, 2017 11:56:05 AM Event level: Information Event: InitializingTime: Friday, June 02, 2017 11:56:06 AM Event level: Information Event: Java Agent Injected aclRetrieve Access Check Logging (ACL) violations or control if such violations will be logged in the Windows Event Log . ACL tracks when processes were unable to load process injection libraries due to insufficient access rights. acl:infoDisplay the following ACL information: Whether it is enabled or disabled Total number of ACL violations that have occurred since the driver was started. The number of access violations will be tracked even if ACL is disabled. C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe acl:infoAccess Check Logging: enabled Total violations: 024 Aternity APM Version11.4.2 Automatic Process Instrumentation on Windows acl:enableEnable logging to the Windows Event Log. When enabled, process injection logs events to the Windows Event Log when a ACL violation is encountered. acl:disableDisable logging to the Windows Event Log. When disabled, process injection will not log events to the Windows Event Log when a ACL violation is encountered. The “acl:info“ command will still report violations, however. java Controls whether the driver will perform Java process injection. The java commands affect the Java framework only. This is in contrast to the “enable“ command, which affects both the Java and .NET frameworks. java:infoDisplay whether Java process injection is enabled: C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe java:info Java Injection: enabled java:enableEnable Java process injection. java:disableDisable Java process injection. java log Controls whether Java process injection will log a message to the Windows Event Log when it loads the profiler library in a process. This logging, when enabled, occurs even if the driver was not used to load the profiler library. The java log:enable command enables the logging and java log:disable disables it. The messages are written to the Windows Application log and look similar to the following: Injected Java Agent: AppInternals. Agent Path: 'C:\Panorama\hedzup\mn\bin\AwProfile.DLL'. Process ID (decimal): 25668. Injected Java Agent: AppInternals. Agent Path: 'C:\Panorama\hedzup\mn\bin\AwProfile64.DLL'. Process ID (decimal): 5244. net Controls whether the driver will perform .NET process injection. The net commands affect the .NET framework only. This is in contrast to the “enable“ command, which affects both the Java and .NET frameworks. Aternity APM Version 11.4.2 25 Automatic Process Instrumentation on Windows net:infoDisplay whether .NET process injection is enabled: C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe net:info .NET Injection: enabled net:enableEnable .NET process injection. net:disableDisable .NET process injection. usdControls or retrieves information about the Unexpected Shutdown Deactivation (USD) safety feature. When USD is enabled and the system experiences an unexpected shutdown, RPID will be deactivated and blocked from restarting. Enabling USD guarantees against the remote possibility of recurring system crashes (also called blue screens or bugchecks) caused by RPID. However, enabling USD will also deactivate RPID after any unexpected shutdown, including after power loss or crashes from any other cause. While enabling USD provides extra safety, it is likely to result in unexpected deactivations of RPID. When USD is enabled and causes deactivation, RPID logs messages visible via the “usd:info“ command:  If RPID determines that the unexpected shutdown does not appear to be because of a system crash, it logs the following message: Due to an unexpected system shutdown, the RPID automatic deactivation feature has deactivated the injection driver. This does not necessarily mean that the injection driver was the cause of the unexpected shutdown but rather this is a safety feature. The RPICTRL.EXE application may be used to activate the injection driver for use (usd:activate) or the automatic deactivation feature can be disabled (usd:disable).  If RPID determines the last unexpected shutdown was caused by a system crash, it logs the following message. Due to an unexpected system shutdown (machine crash), the RPID automatic deactivation feature has deactivated the injection driver. This does not necessarily mean that the injection driver was the cause of the unexpected shutdown but rather this is a safety feature. The RPICTRL.EXE application may be used to activate the injection driver for use (usd:activate) or the automatic deactivation feature can be disabled (usd:disable). In addition, RPID attempts to log USD deactivation entries in the Windows Event Log. These messages may be logged to either the Application or System event log with RPID as the source. USD is disabled by default. The following commands are available for USD: “usd:info“ “usd:enable“ “usd:disable“ “usd:activate“26 Aternity APM Version11.4.2 Automatic Process Instrumentation on Windows usd:infoDisplay information about the current state of USD. For example: C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe usd:infoUnexpected shutdown deactivation: disabled First deactivation: N/A Last deactivation: N/A Deactivation count: 0 Deactivated: no Unexpected Shutdown deactivation – Displays the current status of USD (enabled or disabled)  First deactivation – The first deactivation due to USD occurring on the system  Last deactivation – The last deactivation due to USD occurring on the system  Deactivation count – The number of times RPID has been deactivated due to USD  Deactivated – Displays the current USD deactivation status of RPID (yes or no) This example shows the messages after USD forced deactivation after a shutdown it determined was not a system crash: C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe usd:infoUnexpected shutdown deactivation: enabled First deactivation: Friday, June 02, 2017 3:24:09 PM Last deactivation: Friday, June 02, 2017 3:24:09 PM Deactivation count: 1 Deactivated: yesDue to an unexpected system shutdown, the RPID automatic deactivation feature has deactivated the injection driver. This error could be caused if the system stopped responding, crashed, or lost power. This does not necessarily mean that the injection driver was the cause of the unexpected shutdown but rather this is a safety feature. The injection driver can be activated by using the rpictrl.exe usd:activate command. The USD feature can also be disabled using the rpictrl.exe usd:disable command.------Last System Shutdown Information ------The last system shutdown was not caused by a system crash.It is important to note that this applies only to the last system shutdown and if the system has been rebooted after initial deactivation the cause of the unexpected shutdown will no longer be available.In order to accurately diagnose the cause of deactivation, this should be checked promptly after experiencing the deactivation of the driver. usd:enableEnable USD. usd:disableDisable USD. This is the default.Aternity APM Version 11.4.2 27 Automatic Process Instrumentation on Windows usd:activateActivate RPID for use if it has been deactivated due to USD. This example shows output from the “status“ and “start“ commands after deactivation and then using activate to allow RPID to start: C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe statusRiverbed Process Injection Control Driver Version 0.0.0.0 Copyright 2014-2017. Riverbed Technology. All rights reserved.Start type: system start Status: stopped USD Deactivated: usd:info can be used for more infoC:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe start Error starting process injection. This driver has been blocked from loadingC:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe usd:activateC:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe start Succesfully started process injection. osiControls or retrieves information about the Operating System Incompatibility (OSI) safety feature. RPID will automatically deactivate if an operating system incompatibility is detected. For example, this will happen when the operating system is upgraded to an unsupported version after the agent was installed. This feature is always enabled and cannot be disabled by RPICTRL. RPICTRL can be used to control this feature through the use of the following commands: “osi:info“ “osi:activate“ osi:infoDisplays information about the current state of OSI deactivation. C:\Program Files (x86)\Riverbed\ProcessInjection\bin>rpictrl.exe osi:infoOS Incompatibility deactivation: enabled Last deactivation: N/A Deactivated: no osi:activateActivates RPID for use so that it can once again attempt to start. While it is possible to once again activate RPID for use and once again attempt to check for the incompatibility, the result will most likely be the same. However, it is available in the event that a system change can be made that may affect the OS incompatibility status.28 Aternity APM Version11.4.2 Automatic Process Instrumentation on WindowsAdditional Process Injection FeaturesJava Agent LoggingIn order to better diagnose injection and instrumentation, when the Java process injection library (RPIL) injects the APM profiler library, it logs the following information to the Windows Event Log: Injected Java Agent: APM. Agent Path: 'C:\Panorama\hedzup\mn\bin\AwProfile64.DLL'. Process ID (decimal): 5244.This logging is done within the user-mode component RPIL and will be performed even when the RPID driver is not used. Additional SafetyRPID never executes in Windows Safe Mode. In the event that an issue occurs with the Process Injection driver, an Administrator can boot into Safe Mode and then from an Administrator command prompt disable the driver. Follow these steps:1) Boot the system into Windows Safe Mode (Command Prompt) 2) With an Admin elevated command prompt navigate to the Agent install bin directory3) Within the elevated command prompt, issue the “rpictrl.exe disable” command 4) Reboot5) Upon reboot RPID will be disabled and will not start Aternity APM Version 11.4.2 29 Automatic Process Instrumentation on Windows30 Aternity APM Version11.4.2 CHAPTER 7 Instrumentation Techniques and Troubleshooting This chapter describes general techniques for configuring and troubleshooting issues with instrumentation. For troubleshooting issues specific to Java and .NET Core, see “Troubleshooting Java Instrumentation Issues“ and “Troubleshooting .NET Instrumentation Issues“ respectively.Hotspot DetectionAPM automatically tunes which methods it monitors to keep performance impact to a minimum. This “hotspot” detection insures that APM does not monitor short methods that are called with excessive frequency. JIDA and the dotNet sub-agent dynamically monitor execution time of all methods they monitor. If a method is called many times but has a very short execution time, APM declares it a “hotspot” and automatically disables monitoring. As a result, over time, APM' monitoring of these expensive methods automatically tapers to nothing. Although there are typically only a few hotspot methods in applications (even large applications), testing shows that disabling monitoring of these few methods dramatically reduces APM overhead. APM keeps a record of all the hotspot methods in the <installdir>\Panorama\hedzup\mn\userdata\cache\jida_hotspots and dotnet_hotspots directories. The directory contains zero-length files with the name of each method for which the sub-agents disabled monitoring. For example: [nhapp1:panorama]>ls jida_hotspots/ com.ibm.ejs.container.EJSHome.activateBean com.ibm.ejs.container.EJSHome.createBeanO com.ibm.ejs.container.EJSHome.createWrapper com.ibm.ejs.container.EJSHome.getFinderEntityBeanO com.ibm.ejs.container.EJSHome.getWrapper com.ibm.ejs.container.EJSHome.internalCreateWrapper com.ibm.ejs.container.EJSHome.preFind com.ibm.ejs.container.EJSHome.releaseFinderEntityBeanO com.ibm.websphere.samples.trade.ejb.AccountBean.getDataBean com.ibm.websphere.samples.trade.ejb.ConcreteAccountEJB_b5483e03.ejbActivate . . .Aternity APM Version 11.4.2 31 Instrumentation Techniques and TroubleshootingJIDA and the dotNet sub-agent read the list of hotspot methods in the jida_hotspots directory whenever an application starts, which offers the following benefits: The application gets an immediate performance boost since the sub-agents avoid the overhead of initial hotspot detection. You can copy the <sub-agent>_hotspots directory from a testing environment to production and avoid the overhead of initial detection. You can delete the <sub-agent>_hotspots directory for new versions of your application when you want to force the sub-agents to go through hotspot detection again.Troubleshooting Application Problems Caused by APM Monitoring In some cases, APM monitoring of Java or .NET applications can cause problems, ranging from failure to start to slow performance. These problems fall in to the following categories: Problems caused by initial instrumentation, when the process loads the APM monitoring library at startup. These problems typically keep the application from starting at all.  Problems caused by collection of data for particular classes and methods. These problems typically occur after the application starts. This section describes troubleshooting each category.Problems Caused by Initial Instrumentation Troubleshooting problems caused by initial instrumentation may require assistance from Aternity technical support. Follow these initial steps to disable instrumentation and confirm that the issue is caused by APM: 1) In the “Agent Details Screen“, open the “Edit Processes to Instrument Dialog“ for the processes.2) Clear the Instrument check box to disable instrumentation of the processes and click Save.3) Restart the application. If the application still has the same problem, it was not caused by APM. 4) Re-enable instrumentation of the processes by selecting the Instrument check box in the Edit dialog for the processes. 5) Click the Configuration column for the processes in the Agent Details screen to open the “Define a Configuration Screen“. 6) Enable the “Enable Verbose logging“ option. 7) Restart the application. 8) Look in the sub-agent log file for the processes in <installdir>/Panorama/hedzup/mn/log directory. – Log file names for the dotNet sub-agent start with the prefix DA-dotNet. – Log file names for the JIDA sub-agent start with the prefix DA-JIDA.32 Aternity APM Version 11.4.2 Instrumentation Techniques and TroubleshootingThe messages in the log may provide details on the specific cause of the problem when you contact Aternity technical support.Problems Caused by Collection of DataUse this approach when the problem you are trying to troubleshoot does not prevent the application from running. The following steps do not disable initial instrumentation but narrow the cause of problems to specific classes and methods. 1) Enable diagnostic mode and check the system status page for suggestions on which monitored classes and methods may be causing problems. See “Reviewing ‘no collect hints’ on the System Status Page“ for details.2) Stop collecting any data for instrumented methods by disabling options in the Configuration Settings screen. See “Configuring Minimal Instrumentation“ for details.3) “Incrementally Re-enable Collection“ and restart the application each time. When the application encounters the original problem, you have narrowed the cause to that option.Reviewing ‘no collect hints’ on the System Status Page One cause of problems is when application activity results in an excessive number of calls or ‘call-tree nodes’ in transaction trace files. This can slow down application performance and processing of transaction trace files on the analysis server. The analysis server monitors trace files during processing and detects when the number of calls or call-tree nodes are excessive. When this occurs, and the “Diagnostic Mode“ setting in the “Diagnostic Bundle Screen“ is enabled, the analysis server shows details of the specific methods responsible in the Advanced System Status Information section of the “System Status Screen“. After you enable diagnostic mode, if a trace file contains excessive calls, the tabs shown in the Miscellany subsection include the trace analysis logs tab. Details here include specific methods for which you want to suppress monitoring. For example: 2016-09-01 14:29:33: APPTX_WORKER: WARNING: [/var/lib/appinternals/silo/data/traces/app/DSA-eadb5e979c13/2016/8/31/15/cb88e585c48bb08c4db8c9c 07ef6f56b415ab833bcontinuoustrace@2016-08-31_15.46.28.729-0400@eadb5e979c13@jida@tomcat__opt_tomc at-3.526.apptrace]: number of calls in transaction @ thread 'RMI TCP Connection(idle)' [id=35] hit configured limit of 1,000,000. Additional transactions and calls from that thread will be ignored. Below is the summary of the call stack. Consider suppressing collection of one or few top methods [nocollect] for better transaction partitioning. ------sun.rmi.transport.tcp.TCPTransport$ConnectionHandler::run [1,000,001 nested calls] |sun.rmi.server.UnicastServerRef2::dispatch [8 nested calls] |sun.rmi.server.UnicastServerRef2::dispatch [7 nested calls] |sun.rmi.server.UnicastServerRef2::dispatch [7 nested calls] |sun.rmi.server.UnicastServerRef2::dispatch [7 nested calls] |sun.rmi.server.UnicastServerRef2::dispatch [7 nested calls] |+ 355,219 other calls with 999,964 nested callsCreate Never “Monitoring Filters“ for these methods in the “Define a Configuration Screen“ for the applicable configuration.Aternity APM Version 11.4.2 33 Instrumentation Techniques and TroubleshootingConfiguring Minimal Instrumentation Follow these steps to configure minimal instrumentation: 1) In the “Configurations Screen“, create a new configuration named Minimal Instrumentation or something similar.2) In the Data Collection Settings area, choose the Collection Off option. 3) Click Save at the bottom of the screen.4) Navigate to the “Process List Screen“ (CONFIGURE > Process List), select the application (listed in the Processes to Instrument column), choose Minimal Instrumentation in the list of configurations, and click “Change Configuration“: 5) Wait about 5 minutes to see the effect of changes in the APM interface. You do not need to restart the application.Incrementally Re-enable Collection Incrementally re-enable collection as follows:1) In the Minimal Instrumentation configuration, in the Data Collection Settings area, choose the Custom option.2) In the Monitoring Filters list, delete the single filter (Never *) created as a result of choosing the Collection Off option previously. Presence of this filter overrides other options and prevents enabling them from taking effect. Click the delete icon ( ) to delete the filter:3) Enable an option that was enabled in the original configuration. 4) Click Save at the bottom of the screen.34 Aternity APM Version 11.4.2 Instrumentation Techniques and Troubleshooting5) Restart the application. 6) If the application encounters the original problem, you have narrowed the cause to that option.7) If not, repeat step 3) through step 5) until the application encounters the original problem. When the application encounters the original problem, you have narrowed the cause to that option. Contact Aternity technical support for help in resolving the problem.JIDA Sub-agent: Overriding HTTP Proxy for localhost IP Address If your application server environment specifies a proxy server, you may need to explicitly override use of the proxy server for the localhost IP address. JIDA will not start if the IP address for the localhost host name (typically 127.0.0.1) is directed to a proxy server. If the application server uses an HTTP proxy for that IP address, JIDA cannot connect to the DSA. The following symptoms are characteristic of this problem: Messages similar to the following in the JIDA log files in <installdir>/Panorama/hedzup/mn/log/: Using port 2111 to get a port assignment from the DSA. Cannot connect to DSA. Failure details: java.io.IOException: Service Unavailable The DSA log file (in the same directory) shows no messages about JIDA attempting to connect at that time. Other sub-agents on the same system have no problem connecting to the DSA. To work around this problem, add the following -D system property after other command line modifications in the application server startup configuration. Substitute the correct IP address if necessary: -Dhttp.nonProxyHosts=127.0.0.1Aternity APM Version 11.4.2 35 Instrumentation Techniques and TroubleshootingJBoss Issues The following table lists specific application server versions and known JBoss issues. For the latest list, see knowledge base article KB S27439 (Known Issues with AppInternals Monitoring of Java Application Servers).Server Version IssueJBoss AS 7.0 The same known problem and work-around that apply to “JBoss EAP“ also WildFly applies to JBoss AS/WildFly. 7.1 8.0 (WildFly) 8.2 (WildFly) 9.0 (WildFly) JBoss EAP 6.3 There is a known problem in JBoss EAP that prevents applications from starting. This problem affects applications instrumented with the JIDA sub-agent. See the 6.4 following link for details on the problem: https://access.redhat.com/solutions/312453 See the following link for details on working around the problem: http://stackoverflow.com/questions/24924063/how-to-start-up-jboss-in-windo ws Performance Impact of Enabling End-User-Experience MonitoringEnabling end-user-experience monitoring imposes additional load on the analysis server. This instrumentation option is disabled by default, but administrators can enable it in the applicable configuration settings for a monitored Java or .NET application. When you select this option, JIDA and the dotNet sub-agent add a JavaScript snippet to web pages that they monitor. For every transaction that agents monitor, users’ web browsers send corresponding web page performance data to the analysis server. Testing shows that enabling end-user-experience monitoring has the following separate effects on analysis server performance: It reduces the maximum number of transactions from agents that the analysis server can handle by roughly 30 percent. The web performance data from browsers that the analysis server can handle is limited as follows: – 25 million page loads per hour over HTTP – 750,000 page loads per hour over HTTPS Note—You can avoid the reduced performance of HTTPS by handling SSL termination at a proxy server between the APM analysis server system and external client browsers. In such an environment, you must also specify a different Collection Server Address in the Collector Configuration screen of the APM interface. The online help for the Collector Configuration screen describes this in more detail (see Proxy Considerations for Collecting Traffic from External End Users in that help topic). Keep both these effects in mind when assessing the impact of enabling end-user-experience monitoring. 36 Aternity APM Version 11.4.2 Instrumentation Techniques and TroubleshootingIn the “100 Agents“ test, for example, enabling end-user-experience monitoring reduces the maximum number of transactions from agents that the analysis server can handle by 30 percent, from 6 million per hour to 4 million. Even with a reduced number of transactions from agents, however, the analysis server may not handle the corresponding page-load data. That depends on whether the page loads are over HTTP or HTTPS: If they use HTTP, the analysis server can handle the 4 million page loads per hour (well under the 25 million limit).  If they use HTTPS, the analysis server can handle less than a million page loads per hour and would not tolerate enabling end-user-experience monitoring. The Stop Instrumenting Option for Read-Only Users If enabled by an administrator, a limited Agent configuration menu is displayed to read-only users. This menu only allows access to a limited version of the Agent List and Process List screens. In these screens, the only configuration allowed is to use the Stop Instrumenting button to disable instrumentation for any processes that have already been instrumented. Clicking Stop Instrumenting has the same effect as an administrator clearing the Instrument check box for a process in the Agent Details screen Edit Process to Instrument dialog. For example, on the read-only version of the Process List screen:Note the following: Read-only users can only disable instrumentation. Once it is disabled, the screens do not allow it to be enabled.  Processes must be restarted after disabling instrumentation to actually stop monitoring. Enabling this capability also exposes a limited version of the Agent and Configuration History screens. To enable this capability: For on-premises analysis servers, run the webui_settings ro_allow_inst true CLI command:Aternity APM Version 11.4.2 37 Instrumentation Techniques and Troubleshooting appinternals (config) # webui_settings ro_allow_inst true appinternals (config) # show webui_settings ro_allow_inst (Enable/Disable readonly user's ability to stop instrumentation): true Note—For APM SaaS, contact Aternity APM SaaS operations to enable this capability for your company (also called realm). 38 Aternity APM Version 11.4.2 CHAPTER 8 Deploying Agents in Dynamic Environments This section describes considerations for deploying APM agents in dynamic environments. In these environments, servers that APM monitors can be cloned many times and are created and destroyed frequently. Each instantiation typically has a different host name and IP address. This section includes the following topics related to deploying agents in a dynamic environment: Ability to configure agents so that processes are automatically instrumented when they first start on the agent system. This is important in Docker environments and Platform-as-a-Service (PaaS) environments. . It avoids the need for the agent to connect to an analysis server and for manual configuration of processes in the analysis server interface. See “Configuring Processes Running in PaaS Environments to Be Instrumented on Initial Startup“ for details. Tags that identify servers without relying on their host names and IP addresses. – Typically, you create tags in the analysis server interface, in the “Agent Details Screen“. See the “Tags“ section in that topic for details on using tags. – To configure tags that will take effect when agents first start, edit a file that will be deployed as part of the agent. See “Creating tags with the tags.yaml File for Agents Running in PaaS Environments“ in this section for details.  For Docker containers, install the APM agent on the Docker host system. The single agent installation can monitor any number of containers running on the host. See “Monitoring Docker Containers“ for details.  For Kubernetes and OpenShift, install the APM agent on Kubernetes nodes and configure pods to instrument applications running in them. See “Monitoring Kubernetes and OpenShift Environments When Installing Agents on Individual Nodes“ for details. For Kubernetes, install agent as a DaemonSet. For more information, see “Deploying The Agent on Kubernetes Using a DaemonSet“ For Pivotal Platform environments, download and install a tile using the Pivotal Platform Ops Manager. See “Monitoring VMware Tanzu Applications“ for links with more details.Configuring Processes Running in PaaS Environments to Be Instrumented on Initial Startup In Platform-as-a-Service (PaaS) environments (such as container environments ), applications are deployed before the virtual machines are actually provisioned and started. Aternity APM Version 11.4.2 39 Deploying Agents in Dynamic EnvironmentsIn such environments, deployment scripts are responsible for installing and configuring applications that the virtual machine will host. For APM to monitor such an application, those scripts need to also install agent software and configure processes for instrumentation. This section describes configuring agents so that processes you want to monitor will be automatically instrumented when they first start on the agent system. This configuration requires both the following tasks: Download a configuration file from the analysis server that has the specific configuration settings for the application processes that you want to monitor. See “Creating Configuration Files for Agents Running in PaaS Environments“.  Create a mapping file that specifies a process name and corresponding configuration file. See “Mapping Processes to Configuration Files“. For PaaS environments, write deployment scripts that create configuration and mapping files as described here. This approach to configuration is required only in PaaS environments. However, it can be useful in any environment where you want to avoid initial configuration in the analysis server interface and having to restart processes.Note: After Initial Startup, Make Changes in the Analysis Server Interface  Creating configuration and mapping files as described here is useful only for the initial agent startup. After the initial startup, the agent ignores those files if any user changes the corresponding processes configuration in the “Agent Details Screen“ or configuration settings in the “Define a Configuration Screen“. Creating Configuration Files for Agents Running in PaaS EnvironmentsCreate configurations in the analysis server interface “Define a Configuration Screen“ with the settings you want to apply to processes. Use the Edit menu’s “Download“ option to create configuration files with those settings. The Download operation creates a file using the name <configuration_name>.json: 40 Aternity APM Version 11.4.2 Deploying Agents in Dynamic EnvironmentsCopy the configuration files to the <installdir>/Panorama/hedzup/mn/userdata/config directory on the agent system. As described in “Creating the initial-mapping File“, the config: property in the initial-mapping file specifies a configuration file name. Mapping Processes to Configuration FilesThe initial-mapping file specifies processes that you want to instrument when the agent starts and a corresponding configuration file. Identifying Process Names When the AppInternals agent starts, it discovers processes that are running on the system and assigns them names. The include: property in the initial-mapping file must specify a value that will match this name. Use the following include: values to match commonly-encountered processes:This include: Value Matches Processes For Websphere_* IBM WebSphere Application Server Weblogic_* Oracle WebLogic ServerIIS_* Typical Microsoft Internet Information Services (IIS) application poolsIIS* General entry for IIS. (Simply use IIS* if you do not need to differentiate between IIS variants.) Tomcat* Apache Tomcat BizTalk* Microsoft BizTalk Server JBoss* JBoss application server (both standalone and cluster) For other processes, you can find the process name in the analysis server interface. Look in the Discovered As value for the processes in the “Agent Details Screen“:Aternity APM Version 11.4.2 41 Deploying Agents in Dynamic EnvironmentsCreating the initial-mapping File The agent installation creates a template file named initial-mapping.template file in the <installdir>\Panorama\hedzup\mn\userdata\config directory. The template contains commented-out examples with the process names listed in “Identifying Process Names“. Adapt the file to suit your purposes file and save it as initial-mapping. The initial-mapping file specifies process and configuration file names as include config property pairs: include:processname config:configurationfilename Note the following: You can specify a wildcard at the start or end of a a process name: include:Tomcat* include:*glassfish* Do not put spaces between the include and config properties, the colon ( : ) separator, and the corresponding names. The configuration file name can include the .json file extension, but it is not required. If the file name includes spaces, enclose it in quotation marks ( " ). Otherwise, quotation marks are not allowed. Precede comments with the hash character (number sign) #. Any line starting with # is ignored. The sample file has a single entry that instruments IIS processes. Adapt the sample to suit your purposes. The following example changes the configuration file for IIS processes and adds an entry for the prunsrv process. Both entries specify configuration files created using the analysis server interface and downloaded as described in “Creating Configuration Files for Agents Running in PaaS Environments“: include:IIS_* config:EUE_enabled.json include:prunsrv config:default_config.jsonCreating tags with the tags.yaml File for Agents Running in PaaS EnvironmentsTypically, you create tags in the analysis server interface, in the “Agent Details Screen“. See the “Tags“ section in that topic for details on using tags. This section describes creating tags by editing a file on the agent system. This approach is less convenient than using the analysis server interface, because you must have access to the agent system. But it is useful in the following cases: In dynamic environments where you want tags to take effect when agents first start.  Agents earlier than APM Version 10.9 do not support creating tags in the analysis server interface. Specify tags as key-value pairs in the <installdir>\Panorama\hedzup\mn\userdata\config\tags.yaml file on the agent system. Creating the tags.yaml FileFor APM agents Version 10.9 and later, creating tags in the “Agent Details Screen“ generates the tags.yaml file. 42 Aternity APM Version 11.4.2 Deploying Agents in Dynamic EnvironmentsIn addition, the agent installation creates a template file named tags.yaml.template file in the <installdir>\Panorama\hedzup\mn\userdata\config\ directory. Adapt the file to suit your purposes and save it as tags.yaml. You do not need to restart the agent or (for Version 10.9 agents and later) applications for changes to tags.yaml to take effect. Note: Agents Earlier than Version 10.9 Propagate Tags Less Frequently  For Agents earlier than Version 10.9, changes to tags do not propagate to transaction trace files (the source of data for the Application Map and Search tabs and the Transaction Details window) until the sub-agent starts a new trace file or the application is restarted. tags.yaml Syntax Specify tags using this syntax: key : value Specify multiple values for a key by enclosing a comma-separated list in brackets: key : [ value1, value2, value3 ]Note the following: You must supply a space between the key and the colon ( : ), and the colon and the value. Keys and values can contain letters, numbers, and spaces. Other characters are not allowed. Precede comments with the hash character (number sign) #. Any line starting with # is ignored. Specify an empty string for a value as "".  Keys are case sensitive. Tag1 : foo and tag1 : foo results in separate tags. Changes to the tags.yaml file take effect without restarting the agent. tags.yaml Examples The following examples show some tags. Adapt them to create the tags.yaml file: # Spaces are allowed in both the key and value: logical server : Tier 1 OS : Windows 7 # Create a tag without a value: NoValue : "" # Assign multiple values to a single key: MultiValue : [one, two] # Same value for different keys: Aternity APM Version 11.4.2 43 Deploying Agents in Dynamic EnvironmentsFooKey : Bar FooKey2 : BarHere is how the tags in the previous example would appear in the Tags column of the table in the “Servers Tab“:Monitoring Docker Containers Docker is open-source software that automates the deployment of Linux applications inside software containers. You customize base Docker images to create new images that add applications and make any others changes you want. You then create a container that uses the new image. See the following links for more information on Docker: Docker overview: https://docs.docker.com/engine/understanding-docker/  Introductory tutorial: https://docs.docker.com/engine/tutorials/dockerizing/ Running the Agent on the Docker Host or in ContainersAPM supports two approaches for monitoring applications that run in Docker containers: Install and run the agent on the Docker host. That single installation monitors all the containers you want that start on that host. This is the recommended approach. Use it whenever you have access to the Docker host. See “Recommended: Running the Agent on the Docker Host System“ for details. 44 Aternity APM Version 11.4.2 Deploying Agents in Dynamic Environments Build a Docker image that installs and runs the agent on every container you want to monitor. Use this approach only when you do not have access to the Docker host:Aternity APM Version 11.4.2 45 Deploying Agents in Dynamic EnvironmentsThis approach has the following drawbacks: – Slower container startup, since the agent must start before the monitored application – Additional processes for the agent run in each container – Higher overall overhead, since there potentially many agents run on a single host – Higher disk space usage for each monitored container – Requires rebuilding all monitored Docker images to upgrade agent software See “Alternative: Running the Agent in Docker Containers“ for details.Recommended: Running the Agent on the Docker Host SystemThe recommended approach to monitor applications that run in Docker containers is to install the APM agent on the Docker host system. The single agent installation can monitor any number of containers running on the host. This section describes the following: “One Time Setup on the Docker Host System“ “Starting the Instrumented Container with the Panorama Directory Mounted“ “Troubleshooting“ “How Docker Containers Appear in the APM Interface“One Time Setup on the Docker Host SystemInstalling the Agent This step is the same as for any agent deployment. See the “Agent Installation: Unix-Like Operating Systems“ installation topic for details on installing agent software. Install the agent on a Docker host and specify the analysis server that will report data from containers running from that host. The account that runs the agent must be root (or in the docker group). Otherwise, the agent will not have privileges to populate all the “Docker Container Tags“. The Docker host must be running a supported Linux version. See the System Requirements for specific patches, maintenance levels, and other details of supported operating system versions for installing agent software. The system requirements are available from the APM support page: https://support.riverbed.com/content/support/software/opnet-performance/appinternals-xpert.html After installation, start the agent, as described in “Starting the Agent Software“. Examples in this section assume the agent is installed in /opt.Configuring Processes to Be Instrumented on Container Startup You need to configure files on the Docker host so that processes you want to monitor will be automatically instrumented when their containers start. This configuration requires both the following tasks, described in “Configuring Processes Running in PaaS Environments to Be Instrumented on Initial Startup“ earlier in this section:46 Aternity APM Version 11.4.2 Deploying Agents in Dynamic Environments Download a configuration file from the analysis server that has the specific configuration settings for the application processes that you want to monitor. See “Creating Configuration Files for Agents Running in PaaS Environments“.  Create the initial-mapping file that specifies a process name and corresponding configuration file. See “Mapping Processes to Configuration Files“ for details. For example, here is an initial-mapping file entry for Tomcat processes using the config.json configuration file: include:Tomcat* config:config.jsonCreating an Instrumented Version of the Docker Image To create an instrumented version of the Docker image that you want to monitor, run a script that creates a Dockerfile, then build the instrumented image with that Dockerfile.Running the createDockerfile.sh Script The agent installation creates the script /opt/Panorama/hedzup/mn/bin/createDockerfile.sh on the Docker host. The script generates a Dockerfile in the directory you specify and copies supporting files to subdirectories. The script accepts the following arguments:Argument MeaningTarget directory Directory for the generated Dockerfile and supporting files needed to build the instrumented version of the Docker image. If this directory does not exist, the script will prompt whether it should create it.Image to instrument An existing Docker image to instrument. This script uses this value as the argument to the FROM instruction in the generated Dockerfile. Include a tag value as appropriate. (Docker documentation on FROM) The script does not require this value, but if you omit it, you must edit the generated Dockerfile and supply it. User The user name specified for the Dockerfile USER instruction in the existing base Docker image. This value is optional if the base image does not specify a USER value. If you omit this argument, the script uses a value of root. (Docker documentation on USER)Supply argument values on the command line following the -y argument. If omitted, the script prompts for values. The following example shows creating an instrumented version of a Docker image, zeus.run/app/3tier/tomcat. This example runs the script without arguments on the command line so that it prompts for values. 1) Use the docker images command to see details about the image: [root@11A opt]# cd /opt/Panorama/hedzup/mn/bin [root@11A bin]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE zeus.run/app/3tier/tomcat 2.0 89ee1c3ab841 7 days ago 776.5 MB [root@11A bin]#2) Run the script and respond to the prompts. Note that the image value includes the 2.0 tag. [root@11A bin]# ./createDockerfile.sh  /opt/Panorama/hedzup/mn/bin /opt/Panorama/hedzup/mn/bin Directory to create Dockerfile in: /opt/docker-instrument-3tier-tomcat  Do you want to create: /opt/docker-instrument-3tier-tomcat [y/N]? y  Specify a Docker image and version for the FROM command (optional): zeus.run/app/3tier/tomcat :2.0  If your Docker image has a USER command, enter its username (optional):  ====== Creating: /opt/docker-instrument-3tier-tomcat/Dockerfile Using: FROM zeus.run/app/3tier/tomcat:2.0 Using: USER rootAternity APM Version 11.4.2 47 Deploying Agents in Dynamic Environments To create your instrumented image, use a command like this: docker build -t zeus.run/app/3tier/tomcat-instr:2.0 /opt/docker-instrument-3tier-tomcat ====== [root@11A bin]#3) Confirm that script created the specified directory with a Dockerfile and supporting instr/ subdirectory. [root@11A bin]# ls -al /opt/docker-instrument-3tier-tomcat total 16 drwxr-xr-x 3 root root 4096 Jun 6 12:45 . drwxr-xr-x 6 root root 4096 Jun 8 09:47 .. -rw-r--r-- 1 root root 993 Jun 6 12:45 Dockerfile drwxr-xr-x 4 root root 4096 Jun 6 12:45 instr Here is a parallel example that shows supplying the arguments on the command line following the -y argument: [root@11A bin]# ./createDockerfile.sh -y /opt/docker-instrument-3tier-tomcat-CL zeus.run/app/3tie r/tomcat:2.0 /opt/Panorama/hedzup/mn/bin /opt/Panorama/hedzup/mn/bin  ====== Creating: /opt/docker-instrument-3tier-tomcat-CL/Dockerfile Using: FROM zeus.run/app/3tier/tomcat:2.0 Using: USER root  To create your instrumented image, use a command like this: docker build -t zeus.run/app/3tier/tomcat-instr:2.0 /opt/docker-instrument-3tier-tomcat-CL ======Building the Instrumented Image Use the docker build command to create the instrumented version of the Docker image. Use the -t option to specify the name for new, instrumented image and specify the directory containing the Dockerfile that the script generated (Docker documentation on docker build). The following example uses the original image name with -instr appended. This preserves a recognizable name and it makes it easy to see at a glance if the instrumented version is running. [root@11A bin]# docker build -t zeus.run/app/3tier/tomcat-instr:2.0 /opt/docker-instrument-3tier- tomcat Sending build context to Docker daemon 1.501 MB Step 1/4 : FROM zeus.run/app/3tier/tomcat:2.0 . . . ---> 89ee1c3ab841 Step 2/4 : USER root ---> Running in 96e13c0285e8 ---> 77088499b0da Removing intermediate container 96e13c0285e8 Step 3/4 : COPY ./instr /opt/Panorama/hedzup/mn/bin/./.. ---> bf7a7decdc5c Removing intermediate container ba08d06db1c0  Step 4/4 : RUN /opt/Panorama/hedzup/mn/bin/./../bin/rpictrl.sh install && /opt/Panorama/hedzup/ mn/bin/./../bin/rpictrl.sh enable ---> Running in 2407888d7bfa Process injection already disabled. Successfully uninstalled process injection library. Successfully installed process injection library. Successfully enabled process injection. ---> 0c29f7b28e1048 Aternity APM Version 11.4.2 Deploying Agents in Dynamic EnvironmentsRemoving intermediate container 2407888d7bfa Successfully built 0c29f7b28e10Use the docker images command to confirm that the instrumented image built: [root@11A bin]# docker images REPOSITORY TAG IMAGE ID CREATED SIZ E zeus.run/app/3tier/tomcat-instr 2.0 0c29f7b28e10 12 seconds ago 778 .6 MBAternity APM Version 11.4.2 49 Deploying Agents in Dynamic Environments zeus.run/app/3tier/tomcat 2.0 89ee1c3ab841 7 days ago 776 .5 MBOpening Network Ports for Containers to Access the Docker Host Note: Only Required For Containers That Use The Bridge Network  This section describes steps for opening network ports for containers that use the Docker bridge network. These steps are not required for containers that use the host network (containers that start with the docker run --network=host option).By default, new Docker containers are automatically connected to the Docker bridge network. In this case, monitored processes will not have access to APM ports (Docker documentation on container networking). Without access, no environmental or transaction trace data will be written to the agent on the Docker host. When this happens, the JIDA sub-agent writes log messages similar to the following in <installdir>\Panorama\hedzup\mn\ on the Docker host: java.io.IOException: Trouble getting port from DSA: java.net.NoRouteToHostException: No route to host (Host unreachable)The following iptables commands open required ports on the Docker host. These commands require root access. The ports need to be opened only for the Docker network (for containers, in other words). # Port for initial connection to DSA ptables -I INPUT -p tcp --dport 2111 -i docker0 -j ACCEPT# Port for initial connection to AgentRT iptables -I INPUT -p tcp --dport 7072 -i docker0 -j ACCEPT# Ephemeral ports for subsequent connections iptables -I INPUT -p tcp --dport 33000 -i docker0 -j ACCEPT# Port for initial connection to Profiler Sockets iptables -I INPUT -p tcp --dport 7073 -i docker0 -j ACCEPTThe last command opens a single port for subsequent connections. You must also configure the DSA to use the same fixed port. To configure the DSA to use a fixed port, stop the DSA and edit the file <installdir>\Panorama\hedzup\mn\data\dsa.xml. Change the value of the DsaDaDataPort attribute and restart the DSA. For example: <Attribute name="DsaDaDataPort" value="33000"/>Note: Avoid opening a large range of ephemeral ports because it slows down the startup of containers. Starting the Instrumented Container with the Panorama Directory Mounted Use the docker run command to start the instrumented Docker container. You must add the -v (--volume) option to bind mount the agent installation directory on the host in the container (Docker documentation on docker run). For example: [root@11A zeus]# docker run -d -p 8888:8080 -v /opt/Panorama:/opt/Panorama 50 Aternity APM Version 11.4.2 Deploying Agents in Dynamic Environments zeus.run/app/3tier/tomcat-instr:2.0 79e23b4b2fb81154cec7390d5ff8791f1ba0a90583ea0f025d82e4d57905930fThe docker ps command confirms that the container started: [root@11A zeus]# docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 79e23b4b2fb8 zeus.run/app/3tier/tomcat-instr:2.0 "/bin/sh -c '/opt/sta" 7 seconds ago Up 6 seconds 0.0.0.0:8888->8080/tcp focused_stonebrakerIn Docker swarm mode, specify the --mount type=bind option on the docker service create command (Docker documentation on swarm mode). For example: docker service create \ --mount type=bind,source=/opt/Panorama,destination=/opt/Panorama \ -p 8080:8080 -p 9990:9990 zeus.run/app/3tier/tomcat:2.0 Troubleshooting This section describes problems and the configuration to work around them. Host Firewall Prevents Bridged Network Access to APM Ports This is a common initial setup problem. Use iptables commands on the Docker host to open the ports. See “Opening Network Ports for Containers to Access the Docker Host“ for details.Container Process Runs As Root Instead Of Correct User This occurs if you omit the “User“ argument when “Running the createDockerfile.sh Script“. Edit the Dockerfile to uncomment commands and specify the correct user name. The Dockerfile has instructions embedded in it that make the steps clear: #====== #### Uncomment/edit these lines if your image uses a non-root user #### # RUN groupadd root -g 0 \ && usermod -G root <container-user # USER <container-user>Aternity APM Version 11.4.2 51 Deploying Agents in Dynamic EnvironmentsContainer Cannot Determine the Docker Host IP address Instrumented processes in containers start with the JIDA sub-agent. Normally, JIDA automatically determines the IP address of the Docker host where the agent DSA process is running. However, in secure environments, you may need to manually specify the Docker host IP address. Use one of the following methods. Add the IP address of the Docker host to the initial-mapping file (see “Configuring Processes to Be Instrumented on Container Startup“). This method has the advantage of being specified external to the Docker container. For example: include:Tomcat* config:config.json docker.dsa.host=1.2.3.4 Add the RVBD_DSAHOST environment variable to the docker run command using the -e (--env) option. This method has the advantage of being set in the docker container. For example: docker run -d -p 8888:8080 \ -e RVBD_DSAHOST=1.2.3.4 \ -v /opt/Panorama:/opt/Panorama \ zeus.run/app/3tier/tomcat-instr:2.0 If you use both methods, the RVBD_DSAHOST environment variable takes precedence. Missing Docker Container Tags The account that runs the agent must be root (or in the docker group). Otherwise, the agent will not have privileges to populate all the “Docker Container Tags“. Only the “container id“ tag will be retrieved. How Docker Containers Appear in the APM Interface The agent installed on the Docker host appears in the APM interface agent like any other agent. For instance, it appears as an entry in the “Agent List Screen“ , can be configured in the “Agent Details Screen“, and appears in the “Servers Tab“ like other agents. By comparison, instrumented containers are not full-fledged agents. They do not appear in the Agent List screen and do not have their own Agent Details screen for configuration. The APM interface exposes instrumented containers and their processes as described here. In general, the Docker icon ( ) indicates containers and instrumented processes running in them.Agent Details Screen The“List of Processes to Instrument and Assign JMX and WMI Custom Metric Configurations“ in the Agent Details screen for the Docker host shows instrumented processes running in containers. The Docker icon indicates it is a process running in a container. Click the process to see the Docker image name and container ID: 52 Aternity APM Version 11.4.2 Deploying Agents in Dynamic EnvironmentsApplication Map Tab The “Application Map Tab“ shows Docker hosts as “Server Nodes“ elements, like any other agents. It shows Docker containers within the server node and application instances within the containers:Aternity APM Version 11.4.2 53 Deploying Agents in Dynamic EnvironmentsServers Tab The Servers tab shows Docker containers in the Server column of its table, denoted by the Docker icon and a name formatted as specified in the “Docker Container Display Name Screen“. The Tags column shows special “Docker Container Tags“ that give additional information about the container (as well as any user-defined tags). Click the expander to see all values.54 Aternity APM Version 11.4.2 Deploying Agents in Dynamic EnvironmentsDocker Container Tags Container tags are a special type of “Tags“. They are automatically generated by the agent and provide details about individual Docker containers: Unlike user-defined tags, container tags are not visible in the Tags area of the Agent List or Agent Details screen.  You cannot add or modify container tags.  However, as with in any tags, you can filter on container tags by right-clicking in the “Server“ column of the Servers tab table, and use them in the “Search Tab“ “tag“, “tag.name“, and “tag.value“ search fields.  Special container search fields match the values of some of the container tags (see the following table). The following table summarizes the automatically-generated container tags.Tag Name Description Example container hostname Host name of the Docker container, as specified container hostname=3tier-tomcat-0C4 by the Docker run --hostname argument. (The Docker default for the container hostname is the short container ID.) container id Short Docker container ID container id=7da037511b76 container image Name of the Docker image including the Docker container image=3tier/tomcat-instr:2.0 tag. The “container.image“ search field matches the value of this tag. container name Name of the Docker container, as specified by the container name=condescending_mestorf Docker run --name argument (or automatically created by Docker). The “container.name“ search field matches the value of this tag. container type Type of container. For Docker containers, the container type=Docker value is always Docker. The “container.type“ search field matches the value of this tag. docker hostname Host name of the Docker host where the agent is docker hostname=0C4.internal.zeus installed. The “container.parent“ search field matches the value of this tag. The value of this tag is the same as the docker name tag, without the domain. docker name Fully-qualified domain name (FQDN) of the docker name=0C4.internal.zeus.run Docker host where the agent is installed. Alternative: Running the Agent in Docker Containers As described in “Running the Agent on the Docker Host or in Containers“, this approach is not recommended. Use it if you do not have access to the Docker host. With this approach, as part of the Docker image customizations, you can install and configure the APM Linux agent. The agent will report environmental (operating system) data and monitor instrumented applications that start in containers that use the customized image. The following table lists agent files to include in a Docker build. Typically, you put these files in the same directory as a the Dockerfile for building the image.File Description AppInternals_Agent_<version>_Linux.gz Gzip-compressed agent installation script. Obtain this file from the APM download page. install.properties Response file that specifies agent installation options. As described in “Unattended Agent Installation: Unix-Like Operating Systems“, the -silent argument to the agent installer specifies this file. <configuration_name>.json Configuration file with instrumentation settings for the application processes that you want to monitor. See “Configuring Processes Running in PaaS Environments to Be Instrumented on Initial Startup“ for details on downloading configuration files from the analysis server. Aternity APM Version 11.4.2 55 Deploying Agents in Dynamic EnvironmentsFile Description initial-mapping Mapping file that specifies the processes that you want to instrument when the agent starts and a corresponding configuration file. tags.yaml Tag file with custom identifiers that categorize servers that APM is monitoring. The following example shows an excerpt of a Dockerfile that adds these files and installs the agent using the -silent argument: # Install Appinternals agent WORKDIR /opt ADD appinternals_agent_latest_linux.gz . ADD install.properties . RUN gunzip appinternals_agent_latest_linux.gz RUN chmod +x appinternals_agent_latest_linux RUN ./appinternals_agent_latest_linux -silent install.properties RUN rm -rf appinternals_agent_latest_linux  # Add instrumentation configuration and initial-mapping files  ADD config.json /opt/Panorama/hedzup/mn/userdata/config/ ADD initial-mapping /opt/Panorama/hedzup/mn/userdata/config/ ADD tags.yaml /opt/Panorama/hedzup/mn/userdata/config/ The Docker container must run dsactl start to start the agent when it launches: The agent must start before the application that will be monitored.  Add a 2-second delay between starting the agent and starting the application. This allows the agent to write files required by the JIDA sub-agent when it starts with the application. See the “Controlling Agent Software on Unix-Like Systems“ configuration topic for details on dsactl. See https://docs.docker.com/engine/admin/using_supervisord/ for details on using Supervisor with Docker. After you start a Docker container that uses the customized image, the agent connects to the APM analysis server specified by the O_SI_CUSTOMER_ID option in the response file (install.properties in the previous example). The agent appears in the analysis server interface with the Docker container ID as a agent and server host name:56 Aternity APM Version 11.4.2 Deploying Agents in Dynamic EnvironmentsMonitoring Kubernetes and OpenShift Environments When Installing Agents on Individual NodesKubernetes is an open-source system for automating deployment, scaling and management of containerized applications. OpenShift is a platform as a service (PaaS) built around Docker containers orchestrated and managed by Kubernetes. See the following links for more information: Wikipedia article on Kubernetes  Wikipedia article on OpenShift APM supports monitoring applications running in Kubernetes and OpenShift environments that use Docker as the container runtime. This section describes deploying agents in these environments.Setup on Kubernetes NodesKubernetes nodes are Docker hosts. As with “Monitoring Docker Containers“, you install agents on each Kubernetes node that will deploy pods running the application you want to monitor.Installing the Agent This step is the same as for any agent deployment. See the “Agent Installation: Unix-Like Operating Systems“ installation topic for details on installing agent software. Install the agent on each Kubernetes node and specify the analysis server that will report data from containers running in pods on that node: Typically, the same analysis server will be the target for data from all nodes.  Do not enable automatic instrumentation during the agent installation. (Instead, see “Configuring Pods to Instrument Applications“.) When prompted to enable instrumentation, select no: Enable instrumentation automatically system-wide? [no]: no After the installation is complete, start the agent, as described in “Starting the Agent Software“. Note: Examples in this section assume that the agent is installed in /opt.Changing the Configuration Settings to Use on Pod Startup The agent installation on a Kubernetes node automatically creates a default+config.json configuration file and an initial-mapping file that specifies it in the <installdir>/Panorama/hedzup/mn/userdata/config directory. Any applications that start in pods on that node will use the settings in default+config.json automatically. This behavior reduces the manual intervention required for monitoring applications. Typically, however, you will want to change the configuration to use different settings. To change the configuration, download a configuration file from the analysis server with settings you want and modify the initial-mapping file to specify that configuration file. These steps are described in “Configuring Processes Running in PaaS Environments to Be Instrumented on Initial Startup“ earlier in this Aternity APM Version 11.4.2 57 Deploying Agents in Dynamic Environments topic. Configuring Pods to Instrument Applications In Kubernetes environments, you configure pods to instrument applications that run in them. (This is in contrast to standard Docker environments, where you configure images to instrument individual containers.) Follow the steps described here after installing the agent on Kubernetes nodes. For a pod to instrument applications, it must define environment variables and mount the agent installation directory (opt/Panorama) as a host path volume accessible by the pod. The following table summarizes the environment variables and their values. Except for JAVA_TOOLS_OPTIONS, use the values as shown here (and in “Defining Environment Variables“). For JAVA_TOOLS_OPTIONS, change /opt to the agent installation directory on the Kubernetes node.Environment Variable Value Meaning JAVA_TOOL_OPTIONS -agentpath:/opt/Panorama/hedzup/mn/lib/librpilj64.so Java option to load the AppInternals profiler libraryRVBD_AGENT_FILES 1 Enable agent communication via the profiler sockets process on the agent host RVBD_AGENT_PORT 7073 Port to use for sockets communication RVBD_DSAHOST valueFrom:  Build field to dynamically obtain fieldRef:  agent host name at pod build apiVersion: v1  time fieldPath: status.hostIP There are multiple approaches for configuring pods. The following sections show using the OpenShift web console to modify an existing application’s deployment with the required changes: “Defining Environment Variables“ “Mounting the Agent Installation Directory“ “Confirming Instrumentation“Defining Environment Variables In the OpenShift web console, select the project (in this example, the project is doc-example). In the Applications menu, choose Deployments and then the deployment for the application (in this example, the application is openjdk-app). In the Actions menu, choose Edit YAML:58 Aternity APM Version 11.4.2 Deploying Agents in Dynamic EnvironmentsIn the editing window that opens, find the containers: section. If there is not already an - env: list item under it, paste the bold text in the following example under the containers: section. If there is already an - env: list item, omit the first line of bold text (- env:). Note: Begin the stanza with -env if delineating between each container in your deployment and this is the first property of the container. However, if this is not the first property for the container, use env instead (without the initial dash). spec: containers: - env: - name: JAVA_TOOL_OPTIONS value: '-agentpath:/opt/Panorama/hedzup/mn/lib/librpilj64.so' - name: RVBD_AGENT_FILES value: '1' - name: RVBD_AGENT_PORT value: '7073' - name: RVBD_DSAHOST valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIPAternity APM Version 11.4.2 59 Deploying Agents in Dynamic EnvironmentsIndentation is important in yaml. The editing window performs some validation, and you can use a validator such as https://codebeautify.org/yaml-validator. Click Save in the editing window. Open the Environment tab to confirm that the variables were defined as expected:Mounting the Agent Installation DirectoryYou also edit the yaml configuration to mount the agent installation directory. In the Actions menu, choose Edit YAML and add two stanzas:1) In the spec: section, add the following volumes: specification, at the same level as the containers: section. volumes: - hostPath: path: /opt/Panorama type: '' name: appint2) In the containers: section, add the following volumeMounts: specification, at the same level as the - env: list item: volumeMounts: - mountPath: /opt/Panorama name: appint The following example shows both changes in bold text: spec: volumes: - hostPath: path: /opt/Panorama type: '' name: appint  containers: - env: - name: JAVA_TOOL_OPTIONS value: '-agentpath:/opt/Panorama/hedzup/mn/lib/librpilj64.so' - name: RVBD_AGENT_FILES60 Aternity APM Version 11.4.2 Deploying Agents in Dynamic Environments value: '1' - name: RVBD_AGENT_PORT value: '7073' - name: RVBD_DSAHOST valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP volumeMounts: - mountPath: /opt/Panorama name: appint When you save changes, the order of the sections at a given level in the yaml are rearranged alphabetically. This is expected. Confirming Instrumentation After you change the yaml deployment configuration, OpenShift automatically redeploys pods for the application. In the web console Applications menu, choose Pods. The pod for the application you modified should show a status of Running and at its containers ready:Aternity APM Version 11.4.2 61 Deploying Agents in Dynamic EnvironmentsAfter a few minutes, transaction data should appear in the analysis server interface. In the Search tab, search for the project name (doc-example in this case). server.tag = 'container namespace=doc-example' Upgrading the Agent Installed Directly on Kubernetes NodesUpgrade agent software on each Kubernetes node as described in “Upgrading Agent Software“. You can upgrade without stopping any applications that the agent is monitoring. However, you must restart pods running those applications for them to be instrumented using the upgraded agent version.62 Aternity APM Version 11.4.2 Deploying Agents in Dynamic EnvironmentsDeploying The Agent on Kubernetes Using a DaemonSet In Kubernetes -- unless otherwise configured -- a DaemonSet ensures that all nodes run a copy of a pod. As more nodes are added to the cluster, DaemonSet pods are added to the nodes, and as nodes are removed from the cluster, the corresponding DaemonSet pods are also removed and garbage collected. For more information on DaemonSets, see the Kubernetes documentation. DaemonSets are a very efficient way to deploy a monitoring agent in a Kubernetes environment, since the agent is automatically installed in a DaemonSet pod when a new node is created, then destroyed and garbage collected when a node is removed. A DaemonSet obviates the need to manually deploy and manage the agent on every node in a Kubernetes cluster, which greatly improves scalability and maintainability. Additionally, DaemonSets support deploying in a managed environment, like a Platform as a Service (SaaS), where you do not have access to the individual hosts. The following diagram shows how the AppInternals agent is deployed as a DaemonSet on a Kubernetes cluster:The following sections explain how to install and upgrade an AppInternals agent in a DaemonSet: “Deploying the AppInternals Agent as a DaemonSet (No Helm Charts)“ “Deploying the AppInternals Agent as a DaemonSet (Helm Charts)“ “Upgrading the Agent in a Kubernetes DaemonSet“Before You BeginBefore you deploy the AppInternals agent in a Kubernetes cluster as a DaemonSet, you must ensure that you have the following:Aternity APM Version 11.4.2 63 Deploying Agents in Dynamic Environments Access to a Docker Registry A running Kubernetes cluster Access to the Kubernetes cluster using kubectl A configured Analysis Server that is up and running that the agents can connect to Your Customer ID (if using a SaaS Analysis Server) Your Kubernetes Cluster name (if you would like to tag your containers with this information). An agent configuration specific to your application, which you create with the AppInternals Agent Configuration UI. (This requires a running Analysis Server and is only necessary if you want to specify a specific config for your instrumented applications)Deploying the AppInternals Agent as a DaemonSet (No Helm Charts)To deploy the Appinternals agent as a DaemonSet on a Kubernetes cluster without using a Helm Chart, follow these steps:1. Log in to your local system as a user with access to docker and kubectl.2. Host the Riverbed Docker Image by following these steps:2.1. Download the agent tar.gz file from the Riverbed Support site, where it is referred to as the Aternity APM for Kubernetes Kit. The tar.gz file has a name like the following, rvbd_agent_VERSION.tar.gz, and contains the following files necessary for configuring and deploying a DaemonSet and instrumenting an example Apache Tomcat application:• The Riverbed AppInternals agent image tar file — rvbd_agent_VERSION.tar• A yaml template for the agent DaemonSet — rvbd-agent.yaml • A template for the agent DaemonSet properties — agent-daemonset-env.properties• A template for the instrumentation properties — rvbd-instrumentation-env.properties • A yaml template for setup, including the service account — rvbd-agent-setup.yaml • A yaml application template for Apache Tomcat instrumentation — tomcat-app.yamlNote—The example yaml and properties files in this document are for informational purposes only. Use the template files that ship with the agent to create your site-specific files.2.2. Decompress and untar the agent tar file by entering the following command: # tar zxvf rvbd_agent_VERSION.tar.gz 2.3. Change your working directory to the directory that holds the agent tar file, by entering the following command: # cd rvbd_agent_VERSION 2.4. Host the agent Docker Image. The following is an example of loading the docker image and pushing the image to a private 64 Aternity APM Version 11.4.2 Deploying Agents in Dynamic Environments docker registry where: • <your-registry-server> is your Docker Registry FQDN • <your-name> is your Docker username• <your-pword> is your Docker password # docker login -u <your-name> -p <your-pword> <your-registry-server> # docker load -i rvbd_agent_VERSION.tar# docker tag rvbd_agent:VERSION <your-repository>:VERSION# docker push <your-repository>:VERSIONNote—You will need the location of the image in the registry later when you edit the rvbd-agent.yaml file in step 3.5.3. Configure the Riverbed DaemonSet on the Kubernetes cluster by following these steps:3.1. Log in to your local system as a user with access to docker and kubectl.3.2. (Optional) Although not required, Riverbed recommends that you create a "riverbed" namespace, which will provide better logical isolation between user applications and Riverbed components. The example rvbd-agent-setup.yaml file includes a section to create the "riverbed" namespace. For information on when to use namespaces and how to create them, see Namespaces in the Kubernetes documentation.After you have enabled the riverbed namespace in the rvbd-agent-setup.yaml file, enter the following command to create the namespace in Kubernetes: # kubectl create namespace riverbed 3.3. Create a Kubernetes service account by doing the following:3.3.1 Using the text editor of choice, edit the rvbd-agent-setup.yaml template file to make any necessary changes specific to your site. The file looks like this: rvbd-agent-setup.yamlNote—This example yaml file is for informational purposes only. Cutting and pasting may not preserve formatting, which could result in the yaml file not validating. Additionally, the contents of the actual file could change. Therefore, ensure that you use the template file that ships with the agent. apiVersion: v1 kind: Namespace metadata: name: riverbed--- apiVersion: v1 kind: Secret metadata:Aternity APM Version 11.4.2 65 Deploying Agents in Dynamic Environments name: rvbd-secret namespace: riverbed labels: app: "rvbd"--- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: rvbd-agent rules:- apiGroups:- "" resources:- services- events- endpoints- pods- nodes- componentstatuses verbs:- get- list- watch- apiGroups:- "" resources:- configmaps resourceNames:- rvbdtoken # Kubernetes event collection state- rvbd-leader-election # Leader election token verbs:- get- update- apiGroups: # To create the leader election token- "" resources:66 Aternity APM Version 11.4.2 Deploying Agents in Dynamic Environments- configmaps verbs:- create- nonResourceURLs:- "/version"- "/healthz" verbs:- get- apiGroups: # Kubelet connectivity- "" resources:- nodes/metrics- nodes/spec- nodes/proxy verbs:- get---# You need to use this account for your rvbd-agent daemonset kind: ServiceAccount apiVersion: v1 metadata: name: rvbd-agent namespace: riverbed---# Your admin user needs the same permissions to be able to grant them# Easiest way is to bind your user to the cluster-admin role# See https://cloud.google.com/container-engine/docs/role-based-access-control#setting_up_role-based_ac cess_control apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: rvbd-agent roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: rvbd-agentAternity APM Version 11.4.2 67 Deploying Agents in Dynamic Environments subjects:- kind: ServiceAccount name: rvbd-agent namespace: riverbed# ------EOF ------3.3.2 Run the following kubectl create command to create the service account from the yaml file: # kubectl create -f rvbd-agent-setup.yaml 3.4. Create a Kubernetes ConfigMap by doing the following:3.4.1 Using the text editor of choice, edit the template file agent-daemonset-env.properties and ensure that the values are correct for your site. The file is set up for a SAAS environment, and looks like this: agent-daemonset-env.propertiesRVBD_SAAS_ANALYSIS_SERVER_ENABLED=trueRVBD_ANALYSIS_SERVER_HOST=collector-1.steelcentral.netRVBD_CUSTOMER_ID=customer_idRVBD_ANALYSIS_SERVER_PORT=443 RVBD_MAX_INSTR_LOGS=500#RVBD_APP_CONFIG=new-config#RVBD_LOGICAL_SERVER=tag_value #container_metrics=true# ------EOF ------* If you have an ON-PREM environment, set RVBD_SAAS_ANALYSIS_SERVER_ENABLED=false and set RVBD_ANALYSIS_SERVER_HOST to the FQDN of your Analysis Server.* If you have an alternate configuration file, set RVBD_APP_CONFIG=<name of your alternate config file >. Ensure that the alternate config exists on the Analysis Server before setting it here; otherwise AppInternals will not load a config file.3.4.2 Using the kubectl create command, create a ConfigMap from the properties file, as follows: # kubectl create configmap agent-daemonset-env --from-env-file=./agent-daemonset-env.properties -n riverbed 3.5. Create the Riverbed agent as a Kubernetes DaemonSet by doing the following:3.5.1 Edit the template file rvbd-agent.yaml and ensure that the values are correct for your site, especially the location of the docker registry – returned in step 2.4. The file looks like this: rvbd-agent.yamlNote—This example yaml file is for informational purposes only. Cutting and pasting may not preserve formatting, which could result in the yaml file not validating. Additionally, the contents of the actual file could change. Therefore, ensure that you use the template file that ships with the agent.68 Aternity APM Version 11.4.2 Deploying Agents in Dynamic Environments apiVersion: apps/v1 kind: DaemonSet metadata: name: rvbd-agent namespace: riverbed spec: selector: matchLabels: app: rvbd-agent updateStrategy: type: RollingUpdate # Supported only in Kubernetes version 1.6 or later template: metadata: labels: app: rvbd-agent name: rvbd-agent spec: serviceAccountName: rvbd-agent hostNetwork: true dnsPolicy: ClusterFirstWithHostNet containers:- image: repo/agent:11.4.2.521 name: rvbd-agent securityContext: capabilities: {} privileged: false ports:- containerPort: 2111 hostPort: 2111 name: dsaport protocol: TCP- containerPort: 7071 hostPort: 7071 name: daport protocol: TCP- containerPort: 7072Aternity APM Version 11.4.2 69 Deploying Agents in Dynamic Environments hostPort: 7072 name: agentrtport protocol: TCP- containerPort: 7073 hostPort: 7073 name: profilerport protocol: TCP- containerPort: 7074 hostPort: 7074 name: cmxport protocol: TCP envFrom:- configMapRef: name: agent-daemonset-env env:- name: RVBD_DSAHOST valueFrom: fieldRef: fieldPath: status.hostIP resources: requests: memory: "2G" cpu: "200m" limits: memory: "2G" cpu: "200m" volumeMounts:- name: rvbd-files mountPath: /host/opt/Panorama- name: dockersocket mountPath: /var/run/dockersocket- name: pks-dockersocket mountPath: /var/run/pks-dockersocket- name: procdir mountPath: /host/proc readOnly: true70 Aternity APM Version 11.4.2 Deploying Agents in Dynamic Environments- name: cgroups mountPath: /host/sys/fs/cgroup readOnly: true imagePullSecrets:- name: regcred volumes:- hostPath: path: /opt/Panorama name: rvbd-files- hostPath: path: /var/run/docker.sock name: dockersocket- hostPath: path: /var/vcap/data/sys/run/docker/docker.sock name: pks-dockersocket- hostPath: path: /proc name: procdir- hostPath: path: /sys/fs/cgroup name: cgroups tolerations:- operator: "Exists" effect: "NoSchedule"- operator: "Exists" effect: "NoExecute"#------EOF ------3.5.2 (Optional) If the Docker image is from a private repository, a secret needs to be created for pulling the image in the above yaml file (refer to the imagePullSecrets spec). To create the secret, enter the following command, where <your-registry-server> is your Private Docker Registry FQDN (https://index.docker.io/v1/ for DockerHub), <your-name> is your Docker username, <your-pword> is your Docker password, <your-email> is your Docker email: # kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email> -n riverbed 3.5.3 Run the following command to create the DaemonSet: # kubectl create -f rvbd-agent.yamlAternity APM Version 11.4.2 71 Deploying Agents in Dynamic EnvironmentsNote—During DaemonSet startup, the AppInternals agent is installed as a DaemonSet Agent Pod on each Kubernetes cluster Node, and the Profiler binaries are placed in the hostPath directory to be shared by user Pods. As a result, the DaemonSet must be running before instrumentation can be enabled. 4. Enable Java and .NET Core instrumentation by following these steps:4.1. Using the text editor of choice, edit the template file rvbd-instrumentation-env.properties, and ensure that the values are correct for your site. The file looks like this: rvbd-instrumentation-env.properties# Common propertiesAIX_INSTRUMENT_ALL=1 RVBD_AGENT_FILES=1# JAVA InstrumentationJAVA_TOOL_OPTIONS=-agentpath:/opt/Panorama/hedzup/mn/lib/librpilj64.so # .NET Core InstrumentationCORECLR_ENABLE_PROFILING=1CORECLR_PROFILER={CEBBDDAB-C0A5-4006-9C04-2C3B75DF2F80} CORECLR_PROFILER_PATH=/opt/Panorama/hedzup/mn/lib/libAwDotNetProf64.soDOTNET_ADDITIONAL_DEPS=/opt/Panorama/hedzup/mn/install/dotnet/additionalDeps/Riverbed.Ap pInternals.DotNetCore/DOTNET_SHARED_STORE=/opt/Panorama/hedzup/mn/install/dotnet/store/#------EOF ------4.2. Using the Kubernetes kubectl create command, create a rvbd-instrumentation-env ConfigMap as follows: # kubectl create configmap rvbd-instrumentation-env --from-env-file=./rvbd-instrumentation-env.properties5. Deploy your instrumented application. These steps use Apache Tomcat as an example:5.1. Ensure that the application pod's network policy allows TCP connections from the application pod to the following ports on the DaemonSet agent pod: 2111, 7071, 7072, 7073, 7074.5.2. Understand the variables you need to set in the application yaml file.Three variables in the example Apache Tomcat application yaml file – RVBD_APP_CONFIG, RVBD_APP_INSTANCE, and RVBD_CT_tagname – are of particular importance: • RVBD_APP_CONFIGIf your application has a different config from the default config specified in the agent-daemonset-env.properties file, specify that config in the application’s yaml file. For example: - name: RVBD_APP_CONFIG value: WeatherServiceConfig 72 Aternity APM Version 11.4.2 Deploying Agents in Dynamic EnvironmentsNote—If you change the name of the application's config file or create another config file for the application in the Analysis Server WebUI, you must change the value setting for RVBD_APP_CONFIG using the name of the new config file, and then restart the application.• RVBD_APP_INSTANCEAn instrumented application has a "default instance" that is computed based on the class name and the type of application server, which can be non-intuitive, such as Tomcat__apps_wsse_wss2-D_603. You can use the RVBD_APP_INSTANCE variable to specify a more descriptive instance for the application (such as WeatherService, for example) so it is easier to recognize the application instance in the Process List and Instance tab of the WebUI. For example: - name: RVBD_APP_INSTANCE value: WeatherService Note—Specifying the new instance name in the application yaml file instead of the WebUI, which is node-specific, ensures that every instance of the application deployed in the Kubernetes cluster will report the new instance name in the WebUI.• RVBD_CT_tagnameCreates a “tagname”. The “tagname” shows up as a container tag in the WebUI, displaying the value you set this variable to. Multiple defines of this variable are permitted.5.3. Using the text editor of choice, edit the template file tomcat-app.yaml and ensure that the values are correct for your application. • Specify the volume that contains the instrumentation filesThe instrumentation files need to be mounted from a hostpath, typically /opt/Panorama. You must specify the host volume in the Deployment/spec/template/spec section of the application yaml file.• Specify the mount volume in the instrumented containerEach instrumented container needs to mount the volume that contains the instrumented files. Specify VolumeMount in the Deployment/spec/template/spec/container section of the application yaml file.• Add rvbd-instrumentation-env as the configMapRef for the instrumented container.The instrumented application must add our configmap with the env. variables for instrumentation n the Deployment/spec/template/spec/container section of the application yaml file.• Specify the RVBD_DSAHOST address in the Deployment/spec/template/spec/container/env section of the application yaml file.The tomcat-app.yaml template file looks like this: tomcat-app.yamlNote—This example yaml file is for informational purposes only. Cutting and pasting may not Aternity APM Version 11.4.2 73 Deploying Agents in Dynamic Environments preserve formatting, which could result in the yaml file not validating. Additionally, the contents of the actual file could change. Therefore, ensure that you use the template file that ships with the agent.Note—The bold text indicates sections the user may need to change for site-specific instrumentation, as well as setting RVBD variables.--- kind: List apiVersion: v1 metadata: name: tomcat-app-service-example items:- kind: Service apiVersion: v1 metadata: name: tomcat-app-service spec: selector: name: tomcat-app ports:- protocol: TCP port: 8080 targetPort: 8080 type: NodePort- kind: Deployment apiVersion: extensions/v1beta1 metadata: labels: app: tomcat-app name: tomcat-app spec: replicas: 1 selector: matchLabels: app: tomcat-app template:74 Aternity APM Version 11.4.2 Deploying Agents in Dynamic Environments metadata: labels: app: tomcat-app spec: containers:- image: docker.io/tomcat name: tomcat-app-container ports:- containerPort: 8080 envFrom:- configMapRef: name: rvbd-instrumentation-env env: - name: RVBD_DSAHOST valueFrom: fieldRef: fieldPath: status.hostIP#- name: RVBD_APP_CONFIG# value: new-config#- name: RVBD_APP_INSTANCE# value: MyTomcatApp volumeMounts:- mountPath: /opt/Panorama name: rvbd-files volumes:- hostPath: path: /opt/Panorama name: rvbd-files# ------EOF ------5.4. Deploy your application by entering the following Kubernetes kubectl create command: # kubectl create -f <YOUR_APPLICATION_YAML_FILE>6. The DaemonSet should now be deployed and the Appinternals agent harvesting instrumentation information from the various applications in the Kubernetes Nodes. To verify that the DaemonSet is running, log into the Analysis Server (default UID/PW: admin/riverbed-default), and view the Server and Instances tabs as follows:Aternity APM Version 11.4.2 75 Deploying Agents in Dynamic Environments7. If you defined RVBD_CT_tagname variables, as described in step 5.2., this is how they appear in the WebUI:Deploying the AppInternals Agent as a DaemonSet (Helm Charts) To deploy the Appinternals agent as a DaemonSet on a Kubernetes cluster using Helm Charts, follow these steps: 1. Log in to your local system as a user with access to docker and kubectl.2. Download and install helm. For more information, see the helm documentation here. 76 Aternity APM Version 11.4.2 Deploying Agents in Dynamic Environments3. Host the Riverbed Docker Image by following these steps:3.1. Download the agent tar.gz file from the Riverbed Support site, where it is referred to as the Aternity APM for Kubernetes Kit. The tar.gz file has a name like the following, rvbd_agent_VERSION.tar.gz, and contains the following files necessary for configuring and deploying a DaemonSet with helm and instrumenting an example Apache Tomcat application:• The Riverbed AppInternals agent image tar file — rvbd_agent_VERSION.tar• A yaml application template for Apache Tomcat instrumentation — tomcat-app.yaml• AppInternals-specific Helm Charts that reside in the./helm directoryNote—The example yaml and properties files in this document are for informational purposes only. Use the template files that ship with the agent to create your site-specific files.3.2. Decompress and untar the agent tar file by entering the following command: # tar zxvf rvbd_agent_VERSION.tar.gz 3.3. Change your working directory to the directory that holds the agent tar file, by entering the following command: # cd rvbd_agent_VERSION 3.4. Host the agent Docker Image. The following is an example of loading the docker image and pushing the image to a private docker registry where: • <your-registry-server> is your Docker Registry FQDN • <your-name> is your Docker username• <your-pword> is your Docker password # docker login -u <your-name> -p <your-pword> <your-registry-server># docker load -i rvbd_agent_VERSION.tar# docker tag rvbd_agent:VERSION <your-repository>:VERSION # docker push <your-repository>:VERSIONNote—You will need the location of the image in the registry later when you edit the rvbd-agent.yaml file in step 4.5.4. Configure the Riverbed DaemonSet on the Kubernetes cluster using a Helm Chart by following these steps:4.1. Log in to your local system as a user with access to docker, helm, and kubectl.4.2. (Optional) Although not required, Riverbed recommends that you create a "riverbed" namespace, which will provide better logical isolation between user applications and Riverbed components. For information on when to use namespaces and how to create them, see Namespaces in the Kubernetes documentation.Aternity APM Version 11.4.2 77 Deploying Agents in Dynamic EnvironmentsEnter the following command to create the namespace in Kubernetes: # kubectl create namespace riverbed 4.3. (Optional) If the Docker image is from a private repository, a secret needs to be created for pulling the image in the above yaml file (refer to the imagePullSecrets spec). To create the secret, enter the following command, where <your-registry-server> is your Private Docker Registry FQDN (https://index.docker.io/v1/ for DockerHub), <your-name> is your Docker username, <your-pword> is your Docker password, <your-email> is your Docker email: # kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email> -n riverbedNote—During DaemonSet startup, the AppInternals agent is installed as a DaemonSet Agent Pod on each Kubernetes cluster Node, and the Profiler binaries are placed in the hostPath directory to be shared by user Pods. As a result, the DaemonSet must be running before instrumentation can be enabled. 4.4. Determine the release name before installing the Helm Chart. Note that your working directory should still be the directory that holds the agent tar file: rvbd_agent_VERSION, which also contains the agent’s Helm Chart.4.4.4 The installation of a Helm Chart varies slightly between versions 2 and 3 of Helm. Determine the version of Helm that is installed by entering the following command: # helm version • If Helm v2 is installed, use the following command to specify the “release name”: # helm install --name appint --namespace <Your-namespace> <parameter specification> ./helm • If Helm v3 is installed, use the following format to specify the “release name”: # helm install appint -n <Your-namespace> <parameter specification> ./helm 4.5. Once the release name has been specified, install the Helm Chart using a command like the following, replacing the <variables> with the values specific to your site. Note this example uses a minimum configuration on Helm V3 installed on a SaaS server: # helm install --debug appint -n riverbed \ --set image.name="<SIDECAR_IMAGE>" \ --set image.pullSecrets[0]=regcred \ --set analysisServer.host="<YOUR_ANALYSIS_SERVER>" \ ./helm The following parameters are supported:78 Aternity APM Version 11.4.2 Deploying Agents in Dynamic EnvironmentsYou specify each parameter using the --set key=value[,key=value] argument to the helm install command. For example: # helm install --name appint --namespace riverbed --set analysisServer.customerID ="<YOUR_CUSTOMER_ID>" ./helm Alternatively, you can create a yaml file that specifies the values for the parameters, and then pass the yaml file name as an argument to the helm install command. For example, # helm install --name appint --namespace riverbed -f values.yaml ./helm 4.6. Verify the Helm Chart installation succeeded by entering the following command: # helm list -n riverbed If successful, the command returns output like the following: version.BuildInfo{Version:"v3.0.0", GitCommit:"e29ce2a54e96cd02ccfce88bee4f58bb6e2a28b6", GitTreeState:"clean", GoVersion:"go1.13.4"}5. (Optional) Add a config map to the Application Namespace to enable Java and .NET Core instrumentation.The Agent Daemonset Helm Chart automatically adds a config map (called appint-instrumentation-env) to the default namespace of your cluster. Aternity APM Version 11.4.2 79 Deploying Agents in Dynamic EnvironmentsHowever, if your instrumented application is not running in the default namespace, you need to make the config map available in the application's namespace (<Your App Namespace>) by entering the following command: # kubectl get configmap rvbd-instrumentation-env --namespace=default --export -o yaml | kubectl apply --namespace=<Your App Namespace>-f -6. Deploy your instrumented application. These steps use Apache Tomcat as an example:6.1. Ensure that the application pod's network policy allows TCP connections from the application pod to the following ports on the DaemonSet agent pod: 2111, 7071, 7072, 7073, 70746.2. Understand the variables you need to set in the application yaml file.Three variables in the example Apache Tomcat application yaml file – RVBD_APP_CONFIG, RVBD_APP_INSTANCE, and RVBD_CT_tagname – are of particular importance: • RVBD_APP_CONFIGIf your application has a different config from the default config specified in the agent-daemonset-env.properties file, specify that config in the application’s yaml file. For example: - name: RVBD_APP_CONFIG value: WeatherServiceConfig Note—If you change the name of the application's config file or create another config file for the application in the Analysis Server WebUI, you must change the value setting for RVBD_APP_CONFIG using the name of the new config file, and then restart the application.• RVBD_APP_INSTANCEAn instrumented application has a "default instance" that is computed based on the class name and the type of application server, which can be non-intuitive, such as Tomcat__apps_wsse_wss2-D_603. You can use the RVBD_APP_INSTANCE variable to specify a more descriptive instance for the application (such as WeatherService, for example) so it is easier to recognize the application instance in the Process List and Instance tab of the WebUI. For example: - name: RVBD_APP_INSTANCE value: WeatherService Note—Specifying the new instance name in the application yaml file instead of the WebUI, which is node-specific, ensures that every instance of the application deployed in the Kubernetes cluster will report the new instance name in the WebUI.• RVBD_CT_tagnameCreates a “tagname”. The “tagname” shows up as a container tag in the WebUI, displaying the value you set this variable to. Multiple defines of this variable are permitted.6.3. Using the text editor of choice, edit the template file tomcat-app.yaml and ensure that the values are correct for your application. • Specify the volume that contains the instrumentation filesThe instrumentation files need to be mounted from a hostpath, typically /opt/Panorama. 80 Aternity APM Version 11.4.2 Deploying Agents in Dynamic EnvironmentsYou must specify the host volume in the Deployment/spec/template/spec section of the application yaml file.• Specify the mount volume in the instrumented containerEach instrumented container needs to mount the volume that contains the instrumented files. Specify VolumeMount in the Deployment/spec/template/spec/container section of the application yaml file.• Add appint-instrumentation-env as the configMapRef for the instrumented container.The instrumented application must add our configmap with the env. variables for instrumentation n the Deployment/spec/template/spec/container section of the application yaml file.• Specify the RVBD_DSAHOST address in the Deployment/spec/template/spec/container/env section of the application yaml file.The tomcat-app.yaml template file looks like this: tomcat-app.yamlNote—This example yaml file is for informational purposes only. Cutting and pasting may not preserve formatting, which could result in the yaml file not validating. Additionally, the contents of the actual file could change. Therefore, ensure that you use the template file that ships with the agent.Note—The bold text indicates sections the user may need to change for site-specific instrumentation, as well as setting RVBD variables.--- kind: List apiVersion: v1 metadata: name: tomcat-app-service-example items:- kind: Service apiVersion: v1 metadata: name: tomcat-app-service spec: selector: name: tomcat-app ports:- protocol: TCP port: 8080Aternity APM Version 11.4.2 81 Deploying Agents in Dynamic Environments targetPort: 8080 type: NodePort- kind: Deployment apiVersion: extensions/v1beta1 metadata: labels: app: tomcat-app name: tomcat-app spec: replicas: 1 selector: matchLabels: app: tomcat-app template: metadata: labels: app: tomcat-app spec: containers:- image: docker.io/tomcat name: tomcat-app-container ports:- containerPort: 8080 envFrom:- configMapRef: name: appint-instrumentation-env env: - name: RVBD_DSAHOST valueFrom: fieldRef: fieldPath: status.hostIP#- name: RVBD_APP_CONFIG# value: new-config#- name: RVBD_APP_INSTANCE# value: MyTomcatApp volumeMounts:82 Aternity APM Version 11.4.2 Deploying Agents in Dynamic Environments- mountPath: /opt/Panorama name: rvbd-files volumes:- hostPath: path: /opt/Panorama name: rvbd-files# ------EOF ------6.4. (Optional) Whereas the Helm Chart automatically deploys your application, you can use the following Kubernetes kubectl create command to deploy your application manually: # kubectl create -f <YOUR_APPLICATION_YAML_FILE>7. The DaemonSet should now be deployed and the Appinternals agent harvesting instrumentation information from the various applications in the Kubernetes Nodes. To verify that the DaemonSet is running, log into the Analysis Server (default UID/PW: admin/riverbed-default), and view the Server and Instances tabs as follows:8. If you defined RVBD_CT_tagname variables, as described in step 6.2., this is how they appear in the WebUI:Aternity APM Version 11.4.2 83 Deploying Agents in Dynamic EnvironmentsUpgrading the Agent in a Kubernetes DaemonSet AppInternals supports the dynamic Kubernetes RollingUpdate strategy, which the Kubernetes documentation explains as follows: “With RollingUpdate update strategy, after you update a DaemonSet template, old DaemonSet pods will be killed, and new DaemonSet pods will be created automatically, in a controlled fashion.” To upgrade agents installed as a DaemonSet in a Kubernetes cluster using the RollingUpdate strategy, follow these steps:1. Follow steps step 1. and step 2. in the section “Deploying the AppInternals Agent as a DaemonSet (No Helm Charts)“ to host the new agent image on the Docker Registry.2. Using the text editor of choice, update the rvbd-agent.yaml file to point to the new version of the agent. For more information on working with this file, see step 3.5. in “Deploying the AppInternals Agent as a DaemonSet (No Helm Charts)“.Note—The ConfigMap does not need to be updated. The same settings can be used for the newer agent.3. Initiate the Kubernetes RollingUpdate of the DaemonSet by entering the following command:• Not using Helm Charts # kubectl apply -f rvbd-agent.yaml • Using Helm ChartsFor information on how to upgrade Helm Charts, see the helm documentation here.Monitoring VMware Tanzu Applications VMware Tanzu contains a distribution of Cloud Foundry, an application platform as a service (PaaS). APM supports monitoring applications running in VMware Tanzu environments as well as the ability to monitor the VMs (i.e., Diego cells) upon which the applications are running. 84 Aternity APM Version 11.4.2 Deploying Agents in Dynamic EnvironmentsFor VMware Tanzu deployments, the agent is available directly on the VMware Tanzu Network. Download the agent .pivotal file here: https://network.pivotal.io/products/riverbed-appinternals Documentation for deploying the agent and monitoring applications in VMware Tanzu is also hosted on VMware Tanzu: https://docs.pivotal.io/partners/riverbed-appinternals/ Aternity APM Version 11.4.2 85 Deploying Agents in Dynamic Environments86 Aternity APM Version 11.4.2 CHAPTER 9 Deploying the Agent as an NPM Data Source Note: Available Only with SaaS Analysis Server This feature is available only with the Software as a Service (SaaS) offering of the analysis server (SteelCentral SaaS). It is not available when the analysis server is installed in your environment (“on premises”). This section describes how to deploy the SteelCentral APM agent as a data source for SteelCentral AppResponse Cloud and SteelCentral NetProfiler (via Flow Gateway). In this special-purpose deployment, the agent sends packet data to AppResponse Cloud and SteelFlow traffic data to Flow Gateway. Use the SaaS analysis server (the central APM component for configuration and display) interface to specify the destination for network data. Follow these steps:1) Obtain access to SteelCentral SaaS. One way to obtain access is to register for a free trial. After registering, you receive a user name and password to log in to the SaaS analysis server.2) Log into the SaaS analysis server and download the Linux Latest Agent installer. Use the links in the CONFIGURE > Install Agents screen.3) Install Version 10.16.0 or later of the APM agent on a supported Linux environment. See “Support for Deploying Linux Agents as NPM Data Sources“ in “AppInternals System Requirements: Version 11.4.2“ for more details. For details on installing the agent, see “Agent Installation: Unix-Like Operating Systems“. You must install the agent so it connects to the SaaS analysis server: a) Choose SaaS (option 2) when the agent installer prompts: Choose On-Premises or Saas:1) On-Premises Analysis Server 2) SteelCentral SaaS Analysis Server This option requires a SaaS customer ID. If you do not have a customer ID, do not choose this option, since you cannot complete the installation without it. Obtain a customer ID by registering here: http://www.riverbed.com/steelcentral/steelcentral-saas-early-access-signup.htmlAfter registering, you receive a user name and password to log in to the SaaS analysis server. Your customer ID is on the "Install Agents" screen.Select one of the options above and press enter [1]: 2 Aternity APM Version 11.4.2 87 Deploying the Agent as an NPM Data Source b) The installer prompts for the “Customer ID”. Copy the customer ID from the “Install Agents Screen“ of the SaaS analysis server. Enter your SaaS customer ID copied from the "Install Agents" screen of the SaaS analysis server: cd23f7f7-5b82-4a60-b6f0-xxxxx4) After the installation finishes, start the agent as directed by the installer: ******* Riverbed SteelCentral AppInternals post install steps *******1) Start the Riverbed SteelCentral Agent and Sub-Agents. Log in as 'root' and run: /opt/Panorama/hedzup/mn/bin/agent start[root@nhx2-rh6-4 ~]# /opt/Panorama/hedzup/mn/bin/agent start Starting the Agent... Agent Version: 10.16.0.582 [started] PID: 2752 Name: dsa Command: [./dsa, -d] Up Time: 3 Seconds [started] PID: 2762 Name: npm Command: [./npm_agent] Up Time: 3 Seconds [started] PID: 2770 Name: agentrt Command: [./agentrt] Up Time: 3 Seconds [started] PID: 2773 Name: osda Command: [./os_agent] Up Time: 3 Seconds5) After a few minutes, the newly-installed agent will appear in the CONFIGURE > Agent List screen of the SaaS analysis server. Click the agent to open the “Agent Details Screen“. Scroll down to the Network Data area. in the “Choose Network Data Destination“ area, select the Send options to send data to AppResponse or Flow Gateway (or both). Supply the fully-qualified domain name (FQDN) or IP addresses. Click Save:88 Aternity APM Version11.4.2 Deploying the Agent as an NPM Data SourceNote: Analysis server interface displays may be sparse In a typical APM deployment, the analysis server interface displays data generated by web pages and data from systems and applications running on agents in your environment. However, in the case where the analysis server is used solely for deploying agents as described here, much of this data, including network data, is not available. Tabs in the analysis server interface may display little or no data. This is expected.Aternity APM Version 11.4.2 89 Deploying the Agent as an NPM Data Source90 Aternity APM Version11.4.2 CHAPTER 10 Working with Custom Metrics OverviewAppInternals supports a number of environmental metrics (CPU/Memory/Disk/Network) that it monitors and reports on, and that users can set thresholds and alter notifications against and extract using the Metric REST API. As the following figure illustrates, metrics consist of: An ID that identifies a particular type of metric One or more dimensions that identify the source publishing values to each metricNote: Changing the set of dimensions or the value of a dimension is equivalent to creating a new metric.The list of environmental metrics is described in “Environmental Metrics and Sub-Agents Reference“, and where they appear in the WebUI is explained in “Memory Overview Card“. In addition to environmental metrics, AppInternals also supports the ability to create custom metrics, so that you can collect and analyze data from a variety of non-environmental sources, such as Java Management Extensions (JMX) and Windows Management Instrumentation (WMI). AppInternals also provides an SDK and REST API to collect additional custom metrics for such things as queuing and logs.Note: AppInternals supports several out-of-the-box custom metrics configurations. Once enabled, these OOTB configurations begin collecting metrics that appear in the Custom Metrics tab in the WebUI. For the list of supported out-of-the-box custom metrics configurations and how to enable them, see “Out of the Box Custom Metrics“.The following figure illustrates the Custom Metrics architecture:Aternity APM Version 11.4.2 91 Working with Custom MetricsCustom metrics -- either defined through configuration plugins, the SDK, or the REST API -- are collected by the AppInternals agent and pushed to the Analysis Server where they are aggregated and displayed on the “Custom Metrics Tab“. The following diagram illustrates how Custom Metrics fits into and extends the AppInternals architecture by supporting both out of the box plugins like JMX and WMI, and an SDK and a REST API that allow users to create custom plugins and scripts to extend the Custom Metrics functionality.The extensibility of Custom Metrics is profound. You could, for example, use custom metrics as an adjunct to environmental metrics to determine the exact application or thread that is causing, say, network latency or high cpu utilization, thereby lessening the time for RCA and problem resolution. You could also use custom metrics to monitor a multi-tiered service, like a weather service that collects weather data, as shown in the following figure:92 Aternity APM Version 11.4.2 Working with Custom MetricsVarious weather sensors around the world periodically send data to this Weather Service. The data is routed through an Amazon Elastic Load Balancer (ELB) named 'weather-service-elb’, and behind the load balancer the data is processed by a set of 'weather-collectors'. Each weather-collector is a Tomcat application running inside a container, which runs on an Linux based EC2 Instance with an AppInternals Agent installed. The weather-collectors then write data to a single centralized SQL DB, 'weather-service-db'. Additionally, two separate instances of this service are running in two separate AWS Accounts, beta-v1 and prod-v, in a single AWS Region, us-east-1. Using custom metrics, you could collect the following metrics and monitor the end-to-end health and performance of the Weather Service.Note: This is a hypothetical example. Not all custom metrics listed may be supported. For a list of supported custom metrics, see “Supported Custom Metrics“. CloudWatch ELB > Per-LB Metrics > weather-service-elb : RequestCount // Total requests flowing through the weather-service ELBELB > Per-LB Metrics > weather-service-elb : EstimatedProcessedBytes // Total bytes flowing through the weather-service ELB JMX Plugin JMX > Catalina > GlobalRequestProcessor > requestcount // (rate) requests/sec handled by an individual weather-collectorJMX > Catalina > Servlet > requestCount // requests handled by particular servlet in an individual weather-collectorJMX > Catalina > Executor > activeCount // Thread count of an individual weather-collectorJMX > Catalina > Manager > activeSessions // concurrent sessions of an individual weather-collector Write API (published by a custom client plugin running on the DB server)Aternity APM Version 11.4.2 93 Working with Custom Metrics weather-service-db > table-weather-row-count // Number of rows on the weather table weather-service-db > table-weather-query-count // Number of queries to the weather table WMI Plugin (capturing on Windows host OS metrics on the DB server) WMI > SQLServer:Databases > Log Flushes/sec // Number of log flushes.WMI > System > Processor Queue Length // Number of thread that are waiting for time on the system.The remainder of this chapter discusses the following topics: “Supported Custom Metrics“ “Out of the Box Custom Metrics“ “Configuring Custom Metrics“ “Enabling Custom Metrics Configurations“ “Viewing Custom Metric Definitions“ “Viewing Custom Metrics in the WebUI“ “Disabling the Custom Metrics Engine“ “Working with the Custom Metrics SDK and REST API“Supported Custom MetricsCurrently, AppInternals supports creating custom metrics from the following sources: Java Management Extensions (JMX) Windows Management Instrumentation (WMI) Any and all metrics collected by the CMX SDK and REST APIOut of the Box Custom Metrics AppInternals provides the following out-of-the-box custom metrics configurations that deliver with the Analysis Server so that you can immediately begin collecting custom metrics without having to create them from scratch: JMX –java.lang –JBoss –Tomcat –WebLogic –WebSphere WMI – Microsoft Internet Information Services (IIS) –OS94 Aternity APM Version 11.4.2 Working with Custom Metrics–SQLServerNote: The OOTB custom metrics appear in the main JMX and WMI configuration pages, where they can be cloned and edited. For more information, see “Configuring Custom Metrics“.A best practice is to enable the OOTB custom metrics, determine what metrics you are interested in when you view them in the UI, and then clone the OOTB configuration and create a site-specific configuration that defines only those metrics you are interested in collecting.Enabling OOTB Custom MetricsOOTB custom metrics must be enabled before an agent can begin collecting those metrics for display in the “Custom Metrics Tab“ of the WebUI. For instructions on how to enable OOTB custom metrics, see the following sections: “Enabling Custom Metrics Configurations for JMX“ “Enabling Custom Metrics Configurations for WMI“Configuring Custom MetricsThis section explains how to perform the following tasks: “Configuring Metrics for Java Management Extensions (JMX)“ “Configuring Custom Metrics for Windows Management Interface (WMI)“Before You Begin — Dimensions and Tags Please note the following before configuring custom metrics. Each custom metric is uniquely identified by an id and a set of dimensions. The metric id is a string that identifies a particular type of metric (for example, server-cpu-usage). Dimensions are a set of string key-value pairs that describe the source publishing values to the metric (for example, server-cpu-usage for server=app1 and server-cpu-usage for server=app2).  Metric Ids, dimensions, tag names and values must be <= 80 characters. Changing the set of dimensions or the value of a dimension is equivalent to creating a new metric.  AppInternals performs aggregations across dimensions for metrics having the same id.  When configuring custom metrics in AppInternals the metric id and dimensions are automatically defined based upon what you select for collection. However, you can specify additional dimensions that will provide further views to your data. These might be business or logical descriptions like environment : production or region : west.  When naming custom metrics configurations, AppInternals recommends using a standardized naming convention with uniform prefixes to make it easier to find the configurations in the “Custom Metrics Tab“ of the WebUI and in the list of all metrics configured in the system in the Metrics Definitions page, as explained in “Viewing Custom Metric Definitions“.Aternity APM Version 11.4.2 95 Working with Custom MetricsNote—Adding dimensions requires additional processing and storage on the Analysis Server. Dimensions with many unique values, like session ids or raw URLs, are often poor choices as they result in the creation of a large number of metrics and computations. Consider only adding dimensions with a smaller number of unique values like regions, servers or URL paths instead of raw URLs.  In addition to dimensions, you can add tags to metrics, which are essentially extra metadata that make it easier to search and organize metrics. Tags are not aggregated. The following example illustrates how tags and dimensions are defined:The Analysis Server will perform aggregations on the worker and product dimensions, but not on the workstation tag. You can therefore query on the following: – Values for a specific member in a dimension, as in [ worker : sally, product: foo ] – Aggregated values across dimensions, as in [ worker : *, product: foo ] – Values associated with the workstation tag.Configuring Metrics for Java Management Extensions (JMX) To configure JMX custom metrics, follow these steps:1. Ensure that you have JMX installed on the JVMs you intend to monitor, an Analysis Server up and running, and agents installed on the target systems. For information on installing agents, see “Agent Installation“.Note—Agents do not have to be instrumented to collect JMX metrics, unless the JMX metrics require credentials or the processes are running in a Docker container, in which case the agents must be instrumented to collect JMX metrics.2. To view the MBeans registered with the JMX, use either jconsole or VisualVM. For more information on using these utilities, see the documentation at jconsole and VisualVM, respectively.3. Log in to the Analysis Server as admin. The default admin password is riverbed-default.4. Navigate to the CONFIGURE->CUSTOM METRICS->JMX Configurations page on the WebUI, as follows:96 Aternity APM Version 11.4.2 Working with Custom MetricsThe following JMX Configurations screen appears, which displays all the OOTB and user-created JMX configurations, as well as a link that allows you to create a new JMX configuration:Note—For a list of supported OOTB JMX configurations, see “Supported Custom Metrics“. For information on how to enable the collection of these OOTB JMX metrics by the agent, see “Enabling Custom Metrics Configurations for JMX“.5. From this screen you can do the following:• Download a configuration fileIf you wanted to move a configuration file to another Analysis Server, you would first download it to your system by clicking on the download icon on the entry of the configuration of interest, which downloads the configuration json file.You then copy the configuration file to another Analysis Server, and import it when creating a new JMX configuration by clicking on the Import button on the Import a JMX Configuration section of the Define a JMX Configuration screen, as explained in step 6.2.• Edit an existing JMX configuration (user-created or OOTB)To edit an existing JMX configuration, click on either the clone icon to the right of the configuration entry to copy the configuration recommended for editing OOTB configurations) or the edit icon to Aternity APM Version 11.4.2 97 Working with Custom Metrics edit an existing configuration in place.A fully-populated Define a JMX Configuration screen appears, as follows (truncated in the example figure):Note—This example shows a cloned configuration. If you choose to edit rather than clone the configuration, the Name text box is grayed out.• Create a new JMX configurationTo create a new JMX configuration, click on Define a JMX Configuration and a blank Define a JMX Configuration screen appears, as follows:98 Aternity APM Version 11.4.2 Working with Custom Metrics6. Working with jconsole or VisualVM, create or amend a JMX configuration as follows:6.1. In the Name field, provide a descriptive name for the configuration (unless you are editing an existing configuration, in which case the field will be grayed out).Note—Once you save the configuration you will not be able to change its name, so ensure that you have a consistent naming convention for your JMX configurations before you create them.6.2. (Optional) In the Import a JMX Configuration section, you can import a JMX configuration file by clicking on the Import button. If editing an already-existing configuration, you are warned that importing will overwrite the current file. Typically you would import a configuration file that you downloaded from another Analysis server -- as explained in step 5. -- to populate a new configuration.6.3. In the JMX Metrics section, using jconsole or VisualVM, enter the MBean objects to monitor. Please note the following:• Object names include two parts, separated by a colon -- domain name (which includes dots) and Key properties (name/value pairs separated by commas)• Object names are case sensitive and often have embedded spaces, so Riverbed recommends that you copy and paste them from jconsole or VisualVM• MBean names support the use of the wildcard "*" character, but not regular expressions, as the following example illustrates: "bean":"java.nio:type=BufferPool,name=*" 6.4. In the Attributes section, add one or more attributes for the Object. For each attribute determine if you want AppInternals to track it as a value (unchecked) or as a rate (checked). A value is the default. Check the box to convert value to a rate. A rate is the delta between the current value and Aternity APM Version 11.4.2 99 Working with Custom Metrics last sample interval -- delta/sec, for example. Please note the following:• Attribute names are not case sensitive.• Dot syntax is used for Composite attributes (nested attributes), as the following example illustrates: "attribute":"HeapMemoryUsage.used" • Attribute names support use of the wildcard "*" character, as the following examples illustrate: "attribute":"*""attribute":"HeapMemoryUsage.*""attribute":"*Heap*.* • The following attributes are supported:* Numeric attributes* Numeric attributes nested in composite attributes• The following attributes are not supported:* Attributes containing arrays* Attributes containing tabular data* Strings or other non-numeric type6.5. (Optional) Click the Add JMX Metric link to add additional metrics.6.6. (Optional) Custom metrics support the use of tags, which are key/value pairs, to enhance searching on custom metrics.Note—Tags allow you to include extra metadata on metrics which is helpful for searching or organizing similar metrics. Unlike dimensions, adding a tag does not trigger any additional aggregation on the Analysis Server. A tag is like an index that makes finding a metric easier.You can define a tag here for the custom metric. If you want to define a dimension instead of a tag, check the Use As Dimension check box. For more information, see “Before You Begin — Dimensions and Tags“.6.7. To change the default collection rate of 1 second, click on the triangle to expand the Advanced Options. The following screen appears:Note—Some metrics take up more system resources to collect, and should be collected less often.6.8. When you are satisfied with your JMX Configuration, click the Save button. 100 Aternity APM Version 11.4.2 Working with Custom Metrics7. Once the JMX configuration is created, you must enable an agent to collect those metrics so that they will appear in the WebUI. For more information, see “Enabling Custom Metrics Configurations for JMX“.Configuring Custom Metrics for Windows Management Interface (WMI) To configure custom metrics for WMI, follow these steps:1. Ensure that you have Windows Power Shell installed on the target systems so that you can see the WMI objects and classes you want to use to create custom metrics, an Analysis Server up and running, and agents installed on the target Windows systems. For information on installing agents, see “Agent Installation“.2. Log in to the Analysis Server as admin. The default admin password is riverbed-default.3. Navigate to the CONFIGURE->CUSTOM METRICS->WMI Configurations page on the WebUI, as follows:Note—You can also access the WMI Configurations page from a link in the “Agent Details Screen“.The following WMI Configurations screen appears, which displays all the OOTB and user-created WMI configurations, as well as a link that allows you to create a new WMI configuration:Note—For a list of supported OOTB WMI configurations, see “Supported Custom Metrics“.Aternity APM Version 11.4.2 101 Working with Custom Metrics4. From this screen you can do the following:• Download a configuration fileIf you wanted to move a configuration file to another Analysis Server, you would first download it to your system by clicking on the download icon on the entry of the configuration of interest, which downloads the configuration json file.You then copy the configuration file to another Analysis Server, and import it when creating a new WMI configuration by clinking on the Import button on the Import a WMI Configuration section of the Define a WMI Configuration screen, as explained in step 5.2.• Edit an existing WMI configuration (user-created or OOTB)To edit an existing WMI configuration, click on either the clone icon to the right of the configuration entry to copy the configuration (recommended for editing OOTB configurations) or the edit icon to edit an existing configuration in place.A fully-populated Define a WMI Configuration screen appears, as follows (truncated in the example figure):Note—This example shows a cloned configuration. If you choose to edit rather than clone the configuration, the Name text box is grayed out.102 Aternity APM Version 11.4.2 Working with Custom Metrics• Create a new WMI configurationTo create a new WMI configuration, click on Define a WMI Configuration and a blank Define a WMI Configuration screen appears, as follows:5. Working with Windows Power Shell create or amend a WMI configuration as follows:5.1. In the Name field, provide a descriptive name for the configuration.Note—Once you save the configuration you will not be able to change its name, so ensure that you Aternity APM Version 11.4.2 103 Working with Custom Metrics have a consistent naming convention for your WMI configurations before you create them.5.2. (Optional) In the Import a WMI Configuration section, you can import a WMI configuration file by clicking on the Import button. If editing an already-existing configuration file, you are warned that importing will overwrite the current file. Typically you would import a configuration file that you downloaded from another Analysis server -- as explained in step 4. -- to populate a new configuration.5.3. In the WMI Metrics section, do the following:5.3.1 Enter the WMI Performance Object to monitor. Please note the following:• Object names are case sensitive and often have embedded spaces, so Riverbed recommends that you copy and paste them from Windows Power Shell.• Object names support use of the wildcard "*" character, but not regular expressions.5.3.2 Enter one or more performance counters of interest associated with the Object. A value is the default. Check the box to convert value to a rate. A rate is the delta between the current value and last sample interval -- delta/sec, for example.5.4. In the Instance section, add one or more instances associated with the Performance Object. 5.5. (Optional) Add additional WMI metrics but clicking on the Add WMI Metric link.5.6. (Optional) Custom metrics support the use of tags, which are key/value pairs, to enhance searching on custom metrics.Note—Tags allow you to include extra metadata on metrics which is helpful for searching or organizing similar metrics. Unlike dimensions, adding a tag does not trigger any additional aggregation on the Analysis Server. A tag is like an index that makes finding a metric easier.You can define a tag here for the custom metric. If you want to define a dimension instead of a tag, check the Use As Dimension check box. For more information, see “Before You Begin — Dimensions and Tags“.5.7. To change the default collection rate of 1 second, click on the triangle to expand the Advanced Options. The following screen appears:Note—Some metrics take up more system resources to collect, and should be collected less often.5.8. When you are satisfied with the configuration, click the Save button.6. Once the WMI configuration is created, you must enable an agent to collect those metrics so that they will appear in the WebUI. For more information, see “Enabling Custom Metrics Configurations for WMI“.104 Aternity APM Version 11.4.2 Working with Custom MetricsEnabling Custom Metrics ConfigurationsOnce you have defined a custom metric configuration, you need to enable an agent to collect those metrics on the target system. OOTB custom metrics configurations also need to be enabled in the same way. The following sections explain how to enable custom metrics configurations for JMX and WMI custom metrics.Enabling Custom Metrics Configurations for JMXTo enable both user-defined and OOTB custom metrics for JMX, follow these steps:1. Ensure that you have an agent installed on the target system. For more information on installing agents see “Agent Installation“.Note: Agents do not have to be instrumented to collect JMX metrics, unless the JMX metrics require credentials or the processes are running in a Docker container, in which case the agents must be instrumented to collect JMX metrics.2. Determine if you are enabling one or multiple JMX configurations.• Singular enablement a. Navigate to CONFIGURE->Agent List:The Agent List screen appears:Aternity APM Version 11.4.2 105 Working with Custom Metrics b. Click on the entry for the agent you want to use to collect JMX metrics. The Agent Details screen for that agent appears: c. Click on the edit icon on the right of the screen for each process you want to use to collect JMX metrics. A screen like the following appears: d. Select a JMX configuration for the process to collect metrics against from the Configuration pull-down menu, as follows:106 Aternity APM Version 11.4.2 Working with Custom Metrics e. When you are satisfied with your choice, click the Save button. The agent begins collecting the specified JMX metrics, which can be viewed in the “Custom Metrics Tab“ of the WebUI.• Bulk enablement a. Navigate to CONFIGURE->Process List:The Process List screen appears:Aternity APM Version 11.4.2 107 Working with Custom Metrics b. Check the processes of interest that you want to collect JMX metrics against. c. Select a JMX configuration from the JMX Configuration pull-down menu to associate with the selected processes. The agent begins collecting the specified JMX metrics, which can be viewed in the “Custom Metrics Tab“ of the WebUI.Enabling Custom Metrics Configurations for WMI To enable both user-defined and OOTB custom metrics for WMI, follow these steps:1. Ensure that you have an agent installed on the target Windows system. For more information on installing agents see “Agent Installation“.Note—You must have an agent installed on a Windows system to enable a WMI custom metrics configuration.2. Navigate to CONFIGURE->AGENTS->Agent List:The Agent List screen appears:3. On the Agent List screen, you have two choices for enabling WMI enabling on an agent -- singular or 108 Aternity APM Version 11.4.2 Working with Custom Metrics bulk.• Singular enablement a. Click on the agent that you want to use to collect WMI data, and the Agent Details screen appears for that agent, as follows: b. To enable WMI data collection on that agent, select a WMI configuration from the WMI Data pull-down menu. c. When you are satisfied with your choice, click the Save button.• Bulk enablement a. Select one or more agents from the list of agents on the Agent List screen. b. Select a WMI configuration from the WMI Configuration menu.Aternity APM Version 11.4.2 109 Working with Custom MetricsViewing Custom Metric Definitions You can view the complete list of custom metrics (both user-created and OOTB) by navigating to CONFGIRE->CUSTOM METRICS-Metric Definitions:When you click on Metric Definitions, the Metric Definitions screen appears, as follows, which lists all the metrics defined on the system:You can edit the metric definition by clicking on the edit icon on the far right of the screen. A screen like the following appears:110 Aternity APM Version 11.4.2 Working with Custom MetricsOn this screen you can do the following: Change the display name Change the description of the metric Change the units displayed from the extensive list displayed in the pull-down menu Save or Cancel your changesNote: To edit the configuration and the list of metrics it contains, return to the custom metric configuration page where the custom metric appears, for example the JMX Configuration page.Viewing Custom Metrics in the WebUIOnce you have created custom metrics configurations, you can view them in the Custom Metrics Tab on the WebUI. For more information, see “Custom Metrics Tab“.Aternity APM Version 11.4.2 111 Working with Custom MetricsDisabling the Custom Metrics Engine If you are not using custom metrics, you can disable the Custom Metrics (CMX) Engine on the agent by following these steps:1. Log in to the system where the agent is installed:• On a Linux/Unix agent system, log in as root• On a Windows system, log in as a user who has Administrator privileges2. Change your working directory to the agent’s metadata directory by entering the following command:• On a Linux/Unix system:# cd <AppInternals_Home>/hedzup/mn/metadata • On a Windows system:C:\> <AppInternals_Home>\hedzup\mn\metadata 3. Make a backup copy of process.cmxserver.json by entering the following command:• On a Linux/Unix system:# cp process.cmxserver.json process.cmxserver.json.orig • On a Windows system:C:\> copy process.cmxserver.json process.cmxserver.json.orig4. Using the text editor of choice, open process.cmxserver.json for editing. The file looks like this:5. Change “start”:true to “start”:false. The file should look like this:112 Aternity APM Version 11.4.2 Working with Custom Metrics6. When you are satisfied with the edit, write and quit the file.7. For the change to take effect, you must restart the agents.• For instructions on restarting the agent on a Linux/Unix system, see “Controlling Agent Software on Unix-Like Systems“• For instructions on restarting the agent on a Windows system, see “Controlling Agent Software on Windows Systems“Aternity APM Version 11.4.2 113 Working with Custom MetricsWorking with the Custom Metrics SDK and REST APIThe Custom Metrics SDK and REST API allow users to write plugins to extend the support of custom metrics beyond the out-of-the-box plugins that ship with AppInternals. The SDK is designed to create plugins that process high metric volumes and report metric samples at high frequencies, say once per second. The SDK exposes a JAVA interface and internally maintains a state-full recoverable session with the CMX Server. The REST API, by contrast, is designed to create plugins that have lower metric volumes and report metric samples at lower frequencies, say every minute. The REST API uses standard HTTP/HTTPS connections to communicate with the CMX Server. Therefore it may be used by any library or program that supports an HTTP/HTTPS client, such as curl or Postman. The REST API may be used in either a stateless or state-full manner. Note: If using the REST API a state-full manner, it is the developer's responsibility to maintain state and provide any required recovery mechanism.The following diagram illustrates the SDK and REST API (CMX client) architecture:The JMX and WMI plugins in the figure collect metric data, push that data through the CMX SDK API to the CMX client, which then writes the metric data to the CMX Server. Both the SDK and CMX Server cache dimensions and metric definitions to provide both efficiency and seamless session recovery. The SDK also maintains a state-full session with the CMX Server. The REST API, by contrast, collects metric data and sends it over HTTP/HTTPS to the CMX client, where the metric data is then written to the CMX Server. The REST API may be used in either a stateless or 114 Aternity APM Version 11.4.2 Working with Custom Metrics state-full manner. The metric data that was collected by either the SDK or the REST API is written to the CMX Server, processed, and then written to the persistent CMX Capture file, which is then pushed up to the Analysis Server for aggregation and display.Note: When using the REST API, AppInternals recommends that a session be used. However, since a session requires the ability to modify request headers, if the plugin's HTTP/HTTPS client does not allow modification of request headers the REST API plugin cannot use a session.Writing a CMX Plugin Using the REST API The CMX REST API delivers with the agent and is available here. When writing a CMX plugin using the REST API directly rather than calling the REST API through the CMX SDK, AppInternals recommends -- if possible -- that the plugin be bound by a session. The only reason not to use a session is if the plugin’s HTTP/HTTPS client does not allow modification of request headers. Important: When using the REST API ensure that you buffer data and do not send it to the CMX server every second.Creating a Signed SSL CertificateIf the CMX Server is using secure communications (HTTPS), it will require access to a valid SSL Certificate and SSL Private Key files. By default, the CMX Server creates an unsigned certificate and private key, and writes them to the following files: /<AppInternals_Home>/security/cmx-server.crt /<AppInternals_Home>/security/cmx-server.key If your organization does not allow the use of unsigned certificates or wishes to use a certificate from your preferred SSL vendor, use the following steps to configure the CMX Server to use your certificate:1. Obtain an SSL certificate from your SSL vendor.2. Log into the system where the agent is installed.3. Save the certificate obtained from your SSL vendor to the following directory: /etc/ssl/certs/cmx_ssl.crt4. Save the key file generated when creating the certificate signing request (CSR) to the following directory: /etc/ssl/private/cmx_ssl.keyNote: These instructions use the Ubuntu naming convention and directory placement for the SSL certificate and key files. You may use whatever naming convention and directory placement that corresponds to your operating system requirements or organizational standards.Aternity APM Version 11.4.2 115 Working with Custom Metrics5. Modify the CMX Server configuration file to use the new certificate and key files by following these steps:5.1. Change your working directory to /<AppInternals_Home/ /userdata/plugins/cmx/metadata by entering the following command: # cd /<AppInternals_Home/userdata/plugins/cmx/metadata 5.2. Make a backup copy of configuration-basic.json by entering the following command: # cp configuration-basic.json configuration-basic.json.orig 5.3. Using the text editor of choice, open configuration-basic.json for writing. The file looks like this:5.4. In the comms section of the file, add the sslCertificateFile and sslPrivateKeyFile attributes in double quotation marks followed by a colon, followed by the absolute path to the corresponding certificate and key files that you copied to the server where the agent is installed in steps step 3. and step 4. Enclose the absolute pathnames in double quotes.Note—On Windows systems, escape backslash characters (\) in the absolute pathname with another backslash (\\).The edited file should look like this:116 Aternity APM Version 11.4.2 Working with Custom Metrics5.5. When you are satisfied with your edits, write and quit the file.6. To read in the changes to the CMX Server config file, restart the CMX Server by entering the following commands: # cd /<AppInternals_Home>/bin# agent restart cmxserverRest API Calling Sequence -- Using a SessionThe following diagram illustrates the calling sequence of a CMX plugin using the REST API in a bound session:Aternity APM Version 11.4.2 117 Working with Custom MetricsRest API Calling Sequence -- Not Using a SessionThe following diagram illustrates the calling sequence of a CMX plugin using the REST API without a session:Note: This is not recommended. The plugin should always use a bound session if possible.118 Aternity APM Version 11.4.2 Working with Custom MetricsWriting a CMX Plugin Using the SDK The CMX SDK uses JAVA bindings documented in this section. AppInternals also provides several code examples here. The cmxsdk.jar file is the main jar file for the CMX SDK for Java. This file, as well as all additional required jars, reside in /opt/Panorama/hedzup/mn/lib on the agent. When creating or running a CMX SDK client application, the following jar files must be included in the classpath: cmxsdk.jar  jackson-annotations-2.8.9.jar jackson-core-2.8.9.jar jackson-databind-2.8.9.jar commons-httpclient-3.1.jar commons-codec-1.3.jar commons-logging.jar The following diagram illustrates the calling sequence of a CMX plugin written using the CMX SDK:Aternity APM Version 11.4.2 119 Working with Custom MetricsThe CMX Java SDKThe following are the Public Classes for the CMX Java SDK: “CmxSdkClient Class“ “CmxClientEnvironment Class“ “CmxClientEnvironmentBuilder Class“ “CmxMetricDefinition Class“ “CmxMetricDefinitionBuilderClass“ “CmxMetric Class“ “CmxCallbackLoop Class“ “CmxCallback Interface“ “CmxSdkException Class“120 Aternity APM Version 11.4.2 Working with Custom Metrics “CmxSessionStatus Class“CmxSdkClient Class This is the main class a CMX SDK client will use. It is used to open and close CMX client session. as well as, add metric and dimensions definitions.Modifier and Type Method Description CmxSdkClient(ICmxClientEnvironment env) Constructor. Used to instantiate the CMX SDK client. The env argument is documented in “CmxClientEnvironment Class“. CmxSdkClient(String source) Constructor. Used to instantiate the CMX SDK client with the default environment with the source set to source. void open() Open a session with the CMX Server. If it is determined that a prior session for this source was not properly closed, the session will be properly terminated before opening a new session void close() Close a session with the CMX Server ICmxMetric createMetric(ICmxMetricDefinition md, Create and return an ICmxMetric Map<String, String> dimensions) throws instance from the given metric CmxSdkException definition and dimensions. An ICmxMetric instance is the object used to post metric sample values. If the ICmxMetric instance already exists, it will be returned. ICmxMetric createMetric(ICmxMetricDefinition md, Create and return an ICmxMetric Map<String, String> dimensions, Map<String, instance from the given metric String> tags) throws CmxSdkException definition, dimensions and tags. An ICmxMetric instance is the object used to post metric sample values. If the ICmxMetric instance already exists, it will be returned. ICmxMetricDefinition getMetricDefinition(String mdId) Return the metric definition for the requested id. Null will be returned if the metric definition does not exist. ICmxSessionStatus getStatus() Retrieve the current session status as an ICmxSessionStatus instanceCmxClientEnvironment Class This immutable class encapsulates the environmental the SDK needs to function properly for the implementing plugin. Initially these items include IP of CMX Server (default 127.0.0.1), port CMX Server is listening on (7074), secure comms (false), container ID if running in a container environment, logger (CmxSdkDefaultLogger) and source.Modifier and Type Method Description String getContainerId() Return the Id of the container that is hosting the SDK,Aternity APM Version 11.4.2 121 Working with Custom MetricsString getHost() Return the CMX Server host IP, hostname or FQDN Object getInstance() Return the SDK client instance. The source + instance are used to create a unique identifier for this SDK client. ICmxSdkClientLogger getLogger() Return the logger the SDK is using. The logger must implement ICmxSdkClientLogger interface int getPort() Return the CMX Server listening port String getSource() Returns the client's source value. The source must be unique for each client running on the machine. int getMaxSampleCacheHours() Return the number of samples in hours the SDK will maintain in memory when offline from the CMX Server. int getReportingIntervalSecs() Return the sample reporting interval in seconds. This value is used to convert the maxSamplesCacheHours to an actual # of samples. boolean isSecure() Return a boolean that indicates whether or not comms with the CMX Server are secure122 Aternity APM Version 11.4.2 Working with Custom MetricsCmxClientEnvironmentBuilder Class This builder class is used build a CmxClientEnvironment object.Modifier and Type Method Description ICmxClientEnvironment build() Return an instance of a client environment comprised of the  current values set in this builder . ICmxClientEnvironmentBuilder containerId(ContainerTags.Type Setter used to set the Id of the type, String id) container that is hosting the SDK. Valid values for type are currently ContainerTags.Type.DOCKER and ContainerTags.Type.PCF. static defaultBuilder(String source) Used to create a ICmxClientEnvironmentBuilder CmxClientEnvironment using all default values. The source argument uniquely identifies the CMX SDK user (plugin). Default values are: CMX Server Host (127.0.0.1), CMX Server Port (7074), CMX Server secure comms (false), container Id (null), logger (CmxSdkDefaultLogger). static fromServerBuilder(String source) Used to create a ICmxClientEnvironmentBuilder CmxClientEnvironment using the port and secure from the CMX Server config file (configuration-basic.json) and default values for the remaining fields. The source argument uniquely identifies the CMX SDK user (plugin). Other default values are: CMX Server Host (127.0.0.1), container Id (null), logger (CmxSdkDefaultLogger). In order to use this method, the client should be running in the Panorama/hedzup/mn/bin directory. This "creator" is designed to mimic the pre-CmxClientEnvironment versions of the SDK. ICmxClientEnvironmentBuilder host(String host) Setter used to set the SDK's CMX Server host IP, hostname or FQDN ICmxClientEnvironmentBuilder instance(Object instance) Setter used to set the SDK client's instance. The source + instance values are used to uniquely identify an SDK client within the context of a given OS instance. ICmxClientEnvironmentBuilder logger(ICmxSdkClientLogger Setter used to set the logger the logger) SDK should be using. The logger must implement ICmxSdkClientLogger interface ICmxClientEnvironmentBuilder maxSampleCacheHours(int hours) Setter used to set the number of samples in hours the SDK will maintain in memory when offline from the CMX Server. ICmxClientEnvironmentBuilder port(int port) Setter used to set the SDK's CMX Server listeningAternity APM Version 11.4.2 123 Working with Custom MetricsICmxClientEnvironmentBuilder reportingIntervalSecs Setter used to set the clients's reporting interval in seconds. This value is used to convert the maxSamplesCacheHours to an actual # of samples. ICmxClientEnvironmentBuilder secure(boolean secure) Setter used to set whether or not comms with the CMX Server are secure124 Aternity APM Version 11.4.2 Working with Custom MetricsCmxMetricDefinition Class The CmxMetricDefinition class implements the ICmxMetricDefinition interface. This class is immutable.Modifier and Type Method Description String getDescription() Return the description for this metric. String getDisplayName() Return the display name of this metric. String getId() Return the Id of this metric definition. String getUnits() Return the units for this metric. int getVersion() Return the version of this metric definition. String toString() Return the string representation of this metric definition. Format is id[version|description|display name|units].CmxMetricDefinitionBuilderClass The CmxMetricDefinitionBuilder builder class is used to create a CmxMetricDefinition object. Modifier and Type Method Description CmxMetricDefinitionBuilder(String Constructor. Create an instance source) of a metric definition builder for the given source. ICmxMetricDefinition build() Return an instance of a metric definition comprised of the current values set in this builder. ICmxMetricDefinitionBuilder description(String description) Set the description for a metric definition in this builder. ICmxMetricDefinitionBuilder displayName(String displayName) Set the display name for a metric definition in this builder. ICmxMetricDefinitionBuilder units(String units) Set the units for a metric definition in this builder. ICmxMetricDefinitionBuilder version(int version) Set the version for a metric definition in this builder.CmxMetric Class A CmxMetric class implements the ICmxMetric interface. This is the class that's used to post metric samples.Modifier and Type Method Description ICmxMetric addSample(long tstp, Add a single metric sample (tstp and value) to the double value) metric instance. The tstp value is the number of seconds since the Unix Epoch. Alternatively, it could be the delta in seconds from the previous sample's timestamp, ie 1564410952, 1564410953, 1564410954 or 1564410952, 1, 1. This method returns the current objectAternity APM Version 11.4.2 125 Working with Custom MetricsICmxMetric addSamples(long[] Add multiple metric samples (tstps[] and values[]) tstps, double[] to the metric instance. The tstp values are the values) number of seconds since the Unix Epoch. Alternatively, they could be the delta in seconds from the previous sample's timestamp, ie 1564410952, 1564410953, 1564410954 or 1564410952, 1, 1. This method returns the current object. Map<String, String> getDimensions() Return the dimensions associated with this CmxMetric instance. ICmxMetricMetricDefinition getMetricDefinition() Return the ICmxMetricDefinition instance associated with this CmxMetric instance. Map<String, String> getTags() Return the tags associated with this CmxMetric instance. long[] getTimestamps() Return the current array of sample timestamps for this CmxMetric instance. Note - this array is cleared when samples are successfully sent to the CMX Server. In other words, the array only contains timestamps that have not been sent to the CMX Server. double[] getValues() Return the current array of sample values for this CmxMetric2 instance. Note - this array is cleared when samples are successfully sent to the CMX Server. In other words, the array only contains values that have not been sent to the CMX Server. String toString() Return the string representation of this CmxMetric2 instance. Format is [tstp1,tstp2,...,tstp10,...][value1, value2,...,vakue10,...]. Note - this method only shows the first 10 samples. If there are more than 10 samples, "..." will be appended to the end of each list. String toBigString() Return the string representation of this CmxMetric2 instance. Format is [tstp1,tstp2,...,tstpn][value1, value2,...,value2]. Note - this will show all samples. There could be a performance hit if there are a large number of samples.126 Aternity APM Version 11.4.2 Working with Custom MetricsCmxCallbackLoop ClassModifier and Method Description Type CmxCallbackLoop(CmxCallback Create an instance of CmxCallbackLoop that does a callback callback) to a callback object which implements the CmxCallback interface. The callback interval and "call on start" values are set to their defaults, 1 second and false, respectively. CmxCallbackLoop(CmxCallback Create an instance of CmxCallbackLoop that does a callback callback, int interval) to a callback object at the interval specified by interval. which implements the CmxCallback interface. The "call on start" value is set to its default, false. CmxCallbackLoop(CmxCallback Create an instance of CmxCallbackLoop that does a callback callback, boolean callOnStart) to a callback object with the "call on start" value specified by callOnStart. The callback interval value is set to its default, 1 second. The "call on start" value indicates whether the first callback should be made immediately at the start of the callback loop (true) or after the first interval sleep period has expired (false). CmxCallbackLoop(CmxCallback Create an instance of CmxCallbackLoop that does a callback callback, int interval, boolean to a callback object which implements the CmxCallback callOnStart) interface. The callback interval and "call on start" values are set to interval and callOnStart, respectively. void setCallbackArguments(Object ... Set the list of arguments that will be passed into the callback args) method. This method will throw an IllegalStateException if it is called after the callback loop has started. void setDuration(long seconds) This method allows the caller to set how long, in seconds, the callback loop should run before it exits. This is intended to be used for development and debug purposes only. However, if the developer finds a valid production use for it, more power to him/her. This method will throw an IllegalStateException if it is called after the callback loop has started. Thread start() This method causes the callback loop to start on it's own daemon thread. I probably shouldn't be passing back the Thread object.. void stop() This method is called to gracefully stop the callback loop running on the daemon thread. If the callback loop was started on the current thread (via the run() method), the callback method may call this to shutdown the callback loop. boolean isAlive() Test if the callback loop is running. void waitFor() Wait indefinitely for the callback loop to exit. void waitFor(long millis) Wait at most millis milliseconds for the callback loop to exit. On return, test if thread exited by calling isAlive(). void run() This method causes the callback loop to start on the current thread.CmxCallback InterfaceModifier and Type Method DescriptionAternity APM Version 11.4.2 127 Working with Custom Metrics void processMetricSamples(Object args) Method that gets called by the callback loop at the requested interval (set via CmxCallbackLoop ctor interval arg) to process metric samples. The args argumentis a variable list of objects that may be passed into this method. To set the list of objects to be passed, use the CmxCallbackLoop setCallbackArguments methodprior to starting the callback loop.CmxSdkException Class The CmxSdkException class extends the standard Java Exception class.Modifier and Type Method Description CmxSdkException(String msg) Constructs a CmxSdkException with the specified detail message. CmxSdkException(String msg, Constructs a CmxSdkException with the specified Throwable cause) detail message and cause.CmxSessionStatus Class The CmxSessionStatus class encapsulates various CMX SDK session status' and metricsModifier and Type Method Description int getDimensionSetCount() Return the number of dimensions/tags combinations that have been registered in the current session int getMetricDefinitionCount() Return the number of metric definitions that have been registered in the current session int getMetricCount() Return the number of metric samples that have been generated in the current session long getConnectTime() Return the time the current session opened a connection (session) with the CMX Server. Time is in milliseconds since the epoch. long getLastConnectTime() Return the time the current session last communicated with the CMX Server. Time is in milliseconds since the epoch. long getDisconnectTime() Return the time the current session had a communication loss with the CMX Server. Time is in milliseconds since the epoch. int getConnectCount() Return the number of times the current session was able to open a session with the CMX Server. ICmxSessionStatus.Connection getConnectionStatus() Return the communicate status between the SDK and the CMX Server. Possible values are ONLINE and OFFLINE. ICmxSessionStatus.Session getSessionStatus() Return the session status between the client (plugin) and the SDK. Possible values are OPEN and CLOSED.128 Aternity APM Version 11.4.2 CHAPTER 11 Environmental Metrics and Sub-Agents Reference This section lists environmental metrics collected on systems being monitored by the APM agent. The metrics are collected by different sub-agents. Not all of these metrics appear in the APM interface, and the names may be different. AppInternals also supports Custom Metrics. For more information, see “Working with Custom Metrics“. dotNet Sub-Agent The dotNet sub-agent runs within .NET processes that have been configured to start with it. In addition to collecting transaction trace data, it reports the process-level environmental metrics listed in this section.Memory ManagementMBs consumed / sec % time garbage collecting garbage collections / sec virtual size working set(MB) MBs in useProcess CPUBusy (%) System (%) User (%)Process I/OOther Total (MB) Read Total (MB)Aternity APM Version 11.4.2 129 Environmental Metrics and Sub-Agents ReferenceWrite Total (MB) Total (MB)Process MemoryFaults Page File (MB) Pool Nonpaged (MB) Pool Paged (MB)Threads total runnable threads total waiting threads total threads process idJIDA Sub-AgentThe JIDA sub-agent runs within Java processes that have been configured to start with it. In addition to collecting transaction trace data, it reports the process-level environmental metrics listed in this section. You must explicitly enable JIDA to start with Java processes. The installation topic “Enabling Instrumentation of Java Processes“ describes different configuration approaches in detail. Memory ManagementMBs consumed / sec % time garbage collecting garbage collections / sec % total memory in use Heap MBs in use MBs in use non-Heap MBs in useObjects allocation rate live objects130 Aternity APM Version 11.4.2 Environmental Metrics and Sub-Agents ReferenceProcess CPUBusy (%): Total percentage of CPU consumed by the process. System (%): Percentage of CPU consumed by the process in kernel mode (system calls). User (%): Percentage of CPU consumed by the process in user mode.Process I/OOther Total (MB) Read Total (MB) Write Total (MB) Total (MB)Process MemoryFaults Soft Page File (MB) Pool Nonpaged (MB) Pool Paged (MB) Swaps Working Set (MB)Process NetworkInbound Messages Outbound MessagesThreads threads in group total runnable threads total waiting threads total threadsJMX Sub-Agent The JMX (Java Management Extensions) sub-agent reports process-level metrics for Java processes. The analysis server displays these metrics on the following cards: “CPU Card (Instances Tab)“ “Memory Card (Application Map and Servers Tabs)“Aternity APM Version 11.4.2 131 Environmental Metrics and Sub-Agents Reference “Memory Card (Instances Tab)“ “Threads Card“ The metrics that the JMX sub-agent can collect parallel those available through the Java JConsole monitoring tool. See the following link for more details on JConsole: https://docs.oracle.com/javase/8/docs/technotes/guides/management/jconsole.html The JMX sub-agent is a JMX client that connects to the JVM's JMX server and collects metrics through platform "MXBeans" that implement different management interfaces:  For instrumented processes that start with the “JIDA Sub-Agent“, the JMX sub-agent automatically connects to the JMX server and reports metrics with no further configuration.  For other Java processes, you may have to supply Java system properties on the command line for the sub-agent to connect to the JMX server. See “Configuring Java Processes to Allow the JMX Sub-Agent to Connect“. In addition, the “JMX Data“ section of the “Agent Details Screen“ screen enables or disables this sub-agent for all non-instrumented Java processes on an agent. See the following link for more information on JMX: http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html CPU Process (%) The CPU usage of the process. The Instance CPU Usage option in the “CPU Card (Instances Tab)“ card displays this metric as Busy:MemoryMemory metrics provide details about overall heap and non-heap memory usage. (By comparison, the “Heap Memory Pool“ and “Non-Heap Memory Pool“ metrics provide details about specific memory pools within the heap and non-heap areas.) Heap memory is the runtime data area from which the JVM allocates memory for all class instances and arrays. The heap may be of a fixed or variable size. The garbage collector is an automatic memory management system that reclaims heap memory for objects.  Non-heap memory includes a method area shared among all threads and memory required for the internal processing or optimization for the JVM. It stores per-class structures such as a runtime constant pool, field and method data, and the code for methods and constructors. 132 Aternity APM Version 11.4.2 Environmental Metrics and Sub-Agents ReferenceMemoryUsage Used Total memory used by an application instance. This metric has values for each application instance. It is displayed as Per Instance Memory Usage in the “Memory Card (Instances Tab)“ and Instance Memory Usage in the “Memory Card (Application Map and Servers Tabs)“:Instance Memory Usage is the sum of the "used" item (https://docs.oracle.com/javase/7/docs/api/java/lang/management/MemoryUsage.html#getUsed()) from the HeapMemoryUsage and the NonHeapMemoryUsage attributes of java.lang.management.MemoryUsage. This is true if the JMX plugin is enabled for an instrumented process (the default). The following screen shows these values as displayed in jconsole:Aternity APM Version 11.4.2 133 Environmental Metrics and Sub-Agents ReferenceHeapMemoryUsage used NonHeapMemoryUsage used Total heap and non-heap memory used by an application instance. These metrics are displayed by the Total Memory Usage option in the “Memory Card (Instances Tab)“:The following metrics are not currently displayed in the analysis server:  HeapMemoryUsage free %  HeapMemoryUsage free max %  NonHeapMemoryUsage free %  NonHeapMemoryUsage free max % Memory PoolMemoryPool metrics provide details of Java memory pool usage. A memory pools represents a memory area that the JVM manages. The JVM has at least one memory pool and it may create or remove memory pools during execution. A memory pool can belong either to heap or to non-heap memory (see the “Memory“ metrics topic for more detail on heap and non-heap memory). The specific memory pools available depend on the JVM version and vendor. Heap Memory PoolUsage used Heap memory usage. This metric has values for different memory pools in the application instance. It is displayed by the Heap Usage option in the “Memory Card (Instances Tab)“:Usage used / sec Heap memory usage rate. This metric has values for different memory pools in the application instance. It is displayed by the Heap Usage Rates option in the “Memory Card (Instances Tab)“:134 Aternity APM Version 11.4.2 Environmental Metrics and Sub-Agents ReferenceThe following metrics are not currently displayed in the analysis server:  Usage free %  Usage free max % Non-Heap Memory Pool Usage used Non-heap memory usage. This metric has values for different memory pools in the application instance. It is displayed by the Non-Heap Usage option in the “Memory Card (Instances Tab)“:The following metrics are not currently displayed in the analysis server:  Usage free %  Usage free max % Garbage Collection CollectionCount Total number of garbage collections that have occurred. This metric has values for different garbage collection types. It is displayed by the Garbage Collection Rates option in the “Memory Card (Instances Tab)“:Aternity APM Version 11.4.2 135 Environmental Metrics and Sub-Agents Reference% CollectionTime The percentage of time spent on garbage collection. This metric has values for different garbage collection types. It is displayed by the % Time in Garbage Collection option in the “Memory Card (Instances Tab)“:Threads RunnableThreadCount The average number of threads executing in the JVM. WaitingThreadCount The average number of threads in the JVM that are waiting for another thread to perform some action. This value includes threads in both the WAITING, TIMED_WAITING and BLOCKED states (see https://docs.oracle.com/javase/7/docs/api/java/lang/Thread.State.html for details on thread states). These metrics are displayed as Runnable Threads and Waiting Threads in the “Threads Card“:Configuring Java Processes to Allow the JMX Sub-Agent to ConnectThe JMX sub-agent connects to the JVM's JMX server and collects metrics through platform "MXBeans" that implement different management interfaces. See the following link for more information on JMX: http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html 136 Aternity APM Version 11.4.2 Environmental Metrics and Sub-Agents ReferenceThe sub-agent attempts to connect automatically to local Java processes that are not already instrumented and monitored by the “JIDA Sub-Agent“. If this automatic connection succeeds, no further configuration is required. You will see the process in the “Instances Tab“ with the JMX icon to distinguish it from instrumented Java processes:However, if the automatic local connection does not work, you must provide Java system properties as described in this section and restart the application. If you do not see the process with the JMX icon in the Instances tab, first make sure that the “JMX Data“ setting in the “Agent Details Screen“ is set to ON. Aternity APM Version 11.4.2 137 Environmental Metrics and Sub-Agents ReferenceNext, confirm that the connection failed in the “Agent Details Screen“. In the “Processes to Instrument Details“ area, click the View Report link next to “Instrumentation Health“. Click the Environment tab in the window that opens. If the connection failed, the Jmx Monitoring Status entry will have the message Not Connected - No JMX port available:One way to set system properties is on the Java command line, through -Dproperty=value arguments. However, exactly where you specify the system properties varies by application. It may be in a startup script, user interface, or a configuration file. At a minimum, you need to add the following properties: -Dcom.sun.management.jmxremote.port=0 \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.ssl=false \The jmxremote.port=0 property enables JMX RMI connections on an ephemeral port. (If the application already assigns a a port explicitly, omit the jmxremote.port=0 property. The sub-agent will use the assigned port.) The other properties disable security for RMI connections. 138 Aternity APM Version 11.4.2 Environmental Metrics and Sub-Agents ReferenceFor example, here is a command line that specifies the system properties for the java7app.jar application: java -Dcom.sun.management.jmxremote.port=0 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -jar java7app.jar When the JMX sub-agent successfully makes connects to the process, click the View Report link next to “Instrumentation Health“ for the process. The Environment tab should update to show the connection and port:IBM JVMs require an additional system property (shown in bold) to enable JMX RMI connections: -Dcom.sun.management.jmxremote \ -Dcom.sun.management.jmxremote.port=0 \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.ssl=false \NPM Sub-Agent Note: Available Only with SaaS Analysis Server This feature is available only with the Software as a Service (SaaS) offering of the analysis server (SteelCentral SaaS). It is not available when the analysis server is installed in your environment (“on premises”). The NPM sub-agent worker captures packets visible to the network interfaces on the system where the SteelCentral APM agent is installed. This sub-agent is supported only on Linux systems. A beta version for Windows is also available (see “Windows (Beta)“). The type of data the sub-agent generates and its destination depend on the “Choose Network Data Destination“ settings in the “Network Data (SaaS Analysis Server Only)“ area of the “Agent Details Screen“ screen. There are three options that control the sub-agent’s behavior: Send summarized network data to SteelCentral SaaS With this option , the sub-agent calculates aggregate statistics from packets and writes them to disk. The SaaS analysis server downloads the summary data and displays it on the “Network Tab“. The actual packet data is not available.Aternity APM Version 11.4.2 139 Environmental Metrics and Sub-Agents Reference Send Packets to SteelCentral AppResponse Cloud With this option, the sub-agent sends the actual packet data it captures to the specified AppResponse Cloud instance. This option is available only for agent Version 10.16.0 and later. Send SteelFlow to SteelCentral Flow Gateway With this option, the sub-agent generates SteelFlow records and sends them to SteelCentral Flow Gateway. This option is available only for agent Version 10.16.0 and later, and only for agents installed on Linux.Linux On Linux Systems, as described in “Controlling Agent Software on Unix-Like Systems“, start and stop the NPM sub-agent with the agent start npm and agent stop npm commands. Windows (Beta) Note: Beta Feature Windows support for network data is a beta feature. It is subject to change and changes may not be backwards-compatible with this release. Questions and feedback on the feature should not go through technical support. Please provide feedback, ask questions, and share your experience in the APM area of Aternity’s community forum: https://community.riverbed.com/helpcenter/s/topic/0TO0b000000kLZ6GAM/steelcentral-appinternal sOn Windows systems, the sub-agent requires the Npcap library. The library is not included as part of the agent installation. You can download the Npcap library at https://nmap.org/npcap/. After installing Npcap, restart the AppInternals agent.OS Sub-Agent: Docker As described in “Monitoring Docker Containers“, APM supports two approaches for monitoring applications that run in Docker containers: The recommended approach is to install and run the agent on the Docker host. That single installation monitors all the containers you want that start on that host. This section describes metrics collected with this recommended approach. The other approach is to build a Docker image that installs and runs the agent on every container you want to monitor. With this approach, the metrics are collected by the sub-agent running on the container. See “OS Sub-Agent: Unix-Like Operating Systems“ section for details on those metrics. When the agent is installed on the host system, a Docker variant of the OS sub-agent runs on the host and collects environmental metrics. The sub-agent reads the container files on the host and creates metrics for each container that is monitored. Many of the metrics are the same as those collected by the “OS Sub-Agent: Unix-Like Operating Systems“ on Linux systems. Not all of the metrics are available to the sub-agent on the Docker host. Some metrics are specific to the Docker environment. 140 Aternity APM Version 11.4.2 Environmental Metrics and Sub-Agents ReferenceCPU Metrics Busy (%) User (%) System (%) Memory Metrics Available MBytes (see “Available MBytes“ in “OS Sub-Agent: Unix-Like Operating Systems“ for more detail)Disk Metrics I/O (MBytes/sec) (see “I/O (MBytes/sec)“ in “OS Sub-Agent: Unix-Like Operating Systems“ for more detail) Transfers/sec (see “Transfers /sec“ in “OS Sub-Agent: Unix-Like Operating Systems“ for more detail)Network Metrics I/O (Mbits/sec)SystemCPU Denied (%) This metric is displayed as the “VM CPU Latency Metric“ in the “CPU Card (Application Map and Servers Tabs)“. The sub-agent calculates this value as the percentage of time that the container CPU is throttled. This happens when the container CPU usage reaches the its quota constraint during the sampling period. See the following link to Docker documentation on limiting CPU consumption in containers for more detail: https://docs.docker.com/config/containers/resource_constraints/#cpu OS Sub-Agent: Unix-Like Operating SystemsThis sub-agent runs as the os_agent process. As described in “Controlling Agent Software on Unix-Like Systems“, start and stop the OS sub-agent with the agent start os and agent start os commands. CPUBusy (%) User (%) System (%) Aternity APM Version 11.4.2 141 Environmental Metrics and Sub-Agents ReferenceCPU SubsystemQueue Length Context Switch Load Average Load Average (5 min) Load Average (15 min)MemoryCache Bytes Available on Linux only. The amount of physical memory used as cache. This value is the same as the “cache” column on vmstat for Linux. This metric is displayed as Cache in the Non-System Memory Usage option in the “Memory Overview Card“. Available MBytes Amount of physical memory, including cached memory, that is available on the system. The specific value reported varies by operating system: Solaris: the value of the “free” column reported by the vmstat utility Linux: – Later versions (including Red Hat 7 and later, Ubuntu 14.04 and later): the value of the "available" column reported by the free utility (which is calculated from the “MemAvailable” value in /proc/meminfo) – Earlier versions: estimate the value based on other statistics in /proc/meminfo, /proc/sys/vm/min_free_kbytes, or both This metric is displayed as follows:  As Free Memory in the Non-System Memory Usage and Memory Usage options in the “Memory Overview Card“ As Available Memory in the “Memory Card (Application Map and Servers Tabs)“ Buffer (MB) Available on Linux only. The amount of physical memory ( used as buffers. This value is the same as the “buff” column on vmstat for Linux. This metric is displayed as Buffer in the Non-System Memory Usage option in the “Memory Overview Card“. Swap Used (MB) Available on Linux and Solaris only. Used swap space. This value is the same as the “used” column in the “free –tm” command on Linux, and the “swpd” command in Solaris. This metric is displayed as Swap Used in the Memory Usage option in the “Memory Overview Card“. Swap Free (MB) Available on Linux and Solaris only. Free swap space. This value is the same as the “free” column in the “free –tm” command on Linux, and the “swpd” command in Solaris. 142 Aternity APM Version 11.4.2 Environmental Metrics and Sub-Agents ReferenceThis metric is displayed as Swap Free in the Memory Usage option in the “Memory Overview Card“. Physical (MB) Amount of physical memory in use. This value is the same as the “used” column for “free –t” on Linux. This metric is displayed as Memory Usage in the Memory Usage option in the “Memory Overview Card“. Page Ins Available on AIX and Solaris only. The number of page ins. This value is the same as the “pi” column on vmstat. This metric is displayed as Page Ins in the Page/Swap Rates option in the “Memory Overview Card“. Page Outs Available on AIX and Solaris only. The number of page outs. This value is the same as the “po” column on vmstat. This metric is displayed as Page Outs in the Page/Swap Rates option in the “Memory Overview Card“. Paging Space (%) Available on AIX only. The percentage of swap space in use. This metric is displayed as Paging Space Utilization in the Page/Swap Rates option in the “Memory Overview Card“. Swap (%) Available on Linux and Solaris only. The percentage of swap space in use. This metric is displayed as Swap Utilization in the Page/Swap Rates option in the “Memory Overview Card“. Swap Ins Available on Linux only. The number of swap ins. This value is the same as the “si” column on vmstat. This metric is displayed as Swap Ins in the Page/Swap Rates option in the “Memory Overview Card“. Swap Outs Available on Linux only. The number of swap outs. This value is the same as the “so” column on vmstat. This metric is displayed as Swap Outs in the Page/Swap Rates option in the “Memory Overview Card“. Scan Rate The number of memory pages scanned by the kernel in order to find unused memory to free up. Scan rate is a good indicator of memory pressure on a UNIX system. This value is the same as the “sr” column on vmstat. This metric is displayed as Scan Rate in the Page/Swap Rates option in the “Memory Overview Card“. Physical (%) Percent of physical (real) memory in use. This metric is displayed as % Memory Usage in the “Memory Card (Application Map and Servers Tabs)“The following metrics are not currently displayed in the analysis server:  Virtual (%): Percent of virtual memory in use Virtual (MB): MBytes of virtual memory in useAternity APM Version 11.4.2 143 Environmental Metrics and Sub-Agents Reference Real Total (MB)  Virtual Total (MB) Disk I/O (MBytes/sec) The total (not average) number of megabytes read or written to the device. On Linux, this metric corresponds to the sum of the kB_read and kB_wrtn disk statistics from the iostat utility. This metric is displayed by the Server Disk Read/Write Rates option in the “Disk Card“. Active (%) Percentage of elapsed time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%. On Linux, this value is the same as the %util column in the iostat -x command. This metric is displayed by the Disk I/O Usage option in the “Disk Card“. Queue Length The average queue length for the device over. On Linux, this metric corresponds to the avgqu-sz disk statistic from the iostat utility. This metric is displayed by the Disk Queue option in the “Disk Card“. Average time Per Transfer The average time in seconds that I/O requests to the device waited to be served. On Linux, this includes the time spent by the requests in queue and the time spent servicing them. This metric corresponds to the await disk statistic from the iostat utility. This metric is displayed by the Service Time option in the “Disk Card“. Transfers /sec The number of read and write operations in the last second. On Linux, this metric corresponds to the sum of the r/s and w/s disk statistics from the iostat utility. This metric is displayed by the Disk I/O Transfer Rates option in the “Disk Card“. Logical DiskUsed Space (%) The highest percentage of used space among all the file systems and logical disks on a system. For example, if /usr was 20 percent full, and /opt was 99 percent full, the value for the metric would be 99%. For each server, this metric highlights the disk that has the highest chance of running out of space. This metric is displayed as Disk Space Used in the “Disk Card“ when there are no servers selected. Note that the disk (such as a boot disk) does not need a lot of free space, then high values may not indicate a problem. NetworkI/O Inbound (Mbits/sec) I/O Outbound (Mbits/sec) I/O (Mbits/sec) 144 Aternity APM Version 11.4.2 Environmental Metrics and Sub-Agents ReferenceI/O Inbound (Packets/sec) I/O Outbound (Packets/sec) I/O (Packets/sec) Errors In Errors Out ErrorsSystem (Linux Only)These VMWare metrics are collected on Linux only. CPU Denied (%) This metric is displayed as the “VM CPU Latency Metric“ in the “CPU Card (Application Map and Servers Tabs)“. VM Guest Type VMWare Memory Used (MB) VMWare System Up Time (ms)OS Sub-Agent: Windows This sub-agent runs as the os_agent.exe process and is controlled by the Riverbed SteelCentral AppInternals OS Agent Windows service. It collects metrics that correspond to performance counters available through the Microsoft System Monitor. CLRCLR metrics are displayed by the Heap Usage option in the “Memory Card (Instances Tab)“ . You must select a .NET instance in the table before the Heap Usage option is available:The following metric descriptions are from Microsoft documentation for .NET memory performance counters: https://technet.microsoft.com/en-us/subscriptions/downloads/x2tyfybc%28v=vs.71%29.aspx Aternity APM Version 11.4.2 145 Environmental Metrics and Sub-Agents ReferenceGen0 Heapsize This counter displays the maximum bytes that can be allocated in generation 0 (Gen 0); its does not indicate the current number of bytes allocated in Gen 0. A Gen 0 GC is triggered when the allocations since the last GC exceed this size. The Gen 0 size is tuned by the Garbage Collector and can change during the execution of the application. At the end of a Gen 0 collection the size of the Gen 0 heap is in fact 0 bytes; this counter displays the size (in bytes) of allocations that would trigger the next Gen 0 GC. This counter is updated at the end of a GC; its not updated on every allocation. Gen1 Heapsize This counter displays the current number of bytes in generation 1 (Gen 1); this counter does not display the maximum size of Gen 1. Objects are not directly allocated in this generation; they are promoted from previous Gen 0 GCs. This counter is updated at the end of a GC; its not updated on every allocation. Gen2 Heapsize This counter displays the current number of bytes in generation 2 (Gen 2). Objects are not directly allocated in this generation; they are promoted from Gen 1 during previous Gen 1 GCs. This counter is updated at the end of a GC; its not updated on every allocation. Large Object Heapsize This counter displays the current size of the Large Object Heap in bytes. Objects greater than 20 KBytes are treated as large objects by the Garbage Collector and are directly allocated in a special heap; they are not promoted through the generations. This counter is updated at the end of a GC; its not updated on every allocation.The following metrics are not currently displayed in the analysis server:  AppDomains Loaded GC Time (%) Gen0 Collections Gen1 Collections Gen2 Collections process id Time in JIT (%) Time in Runtime checks (%) Total Heapsize Private BytesCPUBusy (%) User (%) System (%) DPC Time (%) Interrupt (%) Interrupts/sec146 Aternity APM Version 11.4.2 Environmental Metrics and Sub-Agents ReferenceTime Spent on HW (%)CPU SubsystemAverage Queue Length Context Switch Queue LengthDiskI/O (MBytes/sec) I/O Read (MBytes/sec) I/O Write (MBytes/sec) Active (%) Queue Length Disk Reads/sec Average time Per Transfer Bytes/sec Transfers/secLogical DiskUsed Space (%) This metric is parallel with the on Used Space (%) metric on Unix-like operating systems. See “Logical Disk“ in “OS Sub-Agent: Unix-Like Operating Systems“ for more details. Note that on Windows, the metric does not include mapped drives in its calculation. MemoryPhysical (%) Physical (MB) Virtual (%) Virtual (MB) Available Bytes Available MBytes Cache Bytes Committed Bytes In Use (%) Hard Faults Page Faults Page Reads/secAternity APM Version 11.4.2 147 Environmental Metrics and Sub-Agents ReferencePages/sec Pool Nonpaged Bytes Pool Paged Bytes Pool Paged Resident Bytes Soft Faults System Cache Resident Bytes System Code Resident Bytes System Code Total Bytes System Driver Resident Bytes System Driver Total BytesNetworkErrors Errors In Errors Out I/O (Mbits/sec) I/O (Packets/sec) I/O Inbound (Mbits/sec) I/O Inbound (Packets/sec) I/O Outbound (Mbits/sec) I/O Outbound (Packets/sec) Output Queue LengthSystemCPU Denied (%) This metric is displayed as the “VM CPU Latency Metric“ in the “CPU Card (Application Map and Servers Tabs)“. VM Guest Type VMWare Memory Used (MB) Ratio of Interrupts/System Calls Ratio of Page Reads/Disk Reads Total Current Paging File Usage (%) Redirector Current Commands System Calls/sec System Up Time148 Aternity APM Version 11.4.2 Environmental Metrics and Sub-Agents ReferenceProcesses Sub-Agent Note: Available Only with SaaS Analysis Server This feature is available only with the Software as a Service (SaaS) offering of the analysis server (SteelCentral SaaS). It is not available when the analysis server is installed in your environment (“on premises”). The Processes sub-agent monitors individual processes on the system where the APM agent is installed. The “Process Data (SaaS Analysis Server Only)“ section of the “Agent Details Screen“ screen enables or disables this sub-agent on each agent. The analysis server displays metrics from the Processes sub-agent on the Processes tab. See the “Processes Tab“ topic for details on the metrics. This sub-agent is supported only on Linux and Windows systems.Linux On Linux systems, the Processes sub-agent collects data by reading the /proc/<procid>/stats virtual file. See the following link for details on the /proc directory: http://man7.org/linux/man-pages/man5/proc.5.html. As described in “Controlling Agent Software on Unix-Like Systems“, start and stop the Processes sub-agent with the agent start agentrt and agent start agentrt commands. Windows On Windows systems, the Processes sub-agents collects data using Windows Management Instrumentation (WMI) performance counters.Aternity APM Version 11.4.2 149 Environmental Metrics and Sub-Agents Reference150 Aternity APM Version 11.4.2 </div> </article> </div> </div> </div> <script type="text/javascript" async crossorigin="anonymous" src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-8519364510543070"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.6.1/jquery.min.js" crossorigin="anonymous" referrerpolicy="no-referrer"></script> <script> var docId = 'a52c12445d2ee2dcf32fdf669652f9ab'; var endPage = 1; var totalPage = 490; var pfLoading = false; window.addEventListener('scroll', function () { if (pfLoading) return; var $now = $('.article-imgview .pf').eq(endPage - 1); if (document.documentElement.scrollTop + $(window).height() > $now.offset().top) { pfLoading = true; endPage++; if (endPage > totalPage) return; var imgEle = new Image(); var imgsrc = "//data.docslib.org/img/a52c12445d2ee2dcf32fdf669652f9ab-" + endPage + (endPage > 3 ? ".jpg" : ".webp"); imgEle.src = imgsrc; var $imgLoad = $('<div class="pf" id="pf' + endPage + '"><img src="/loading.gif"></div>'); $('.article-imgview').append($imgLoad); imgEle.addEventListener('load', function () { $imgLoad.find('img').attr('src', imgsrc); pfLoading = false }); if (endPage < 7) { adcall('pf' + endPage); } } }, { passive: true }); </script> <script> var sc_project = 11552861; var sc_invisible = 1; var sc_security = "b956b151"; </script> <script src="https://www.statcounter.com/counter/counter.js" async></script> </html><script data-cfasync="false" src="/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js"></script>