Setup Guide:
SAP Convergent Mediation by DigitalRoute Application Operation Integration (Solution Manager 7.2)
Version: 3.2 Date: Jan 20th, 2021
Contents 1 PREREQUSITES ...... 3 1.1 Software Requirement ...... 3 1.2 SAP Solution Manager Requirement ...... 3 1.3 Related SAP Notes ...... 4 2 PURPOSE ...... 4 3 LANDSCAPE INTEGRATION ...... 4 3.1 Understand SAP CM Landscape Modeling ...... 4 3.2 Installation Best Practice and Naming Conventions...... 5 3.3 SAP Diagnostic Agent Installation ...... 6 3.4 SLD Data Supplier Integration...... 6 4 SUPPORTABILITY CONTENT PREPRATION ...... 11 4.1 SAP Supportability Package for SAP CM ...... 11 4.1.1 Supportability Package Content ...... 11 4.1.2 Supportability Package Import Procedure ...... 12 4.1.3 Post Import Customizing Actions ...... 14 4.2 RCA Template for SAP CM ...... 16 4.3 Logging Format for SAP CM ...... 18 5 MANAGED SYSTEM SETUP AND CUSTOMIZING ...... 18 5.1 Managed System Setup ...... 19 5.2 Customize Monitoring Parameter for SAPCM Technical Instance ...... 22 6 INTROSCOPE INSTRUMENT ACTIVATION ...... 23 6.1 Introscope Bytecode Agent configuration ...... 23 6.2 Introscope Management Module configuration...... 25 7 ROOT CAUSE ANALYSIS INTEGRATION ...... 26 7.1 Change Reporting and CCDB Stores ...... 26 7.2 Logviewer...... 27 7.3 Exception Management ...... 28 8 SYSTEM MONITORING INTEGRATION ...... 31
2 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION PREREQUSITES
Software Requirement
SAP Convergent Mediation product is a SAP partner product developed by company DigitalRoute in Sweden. It is a core component in SAP BRIM solution and many Telecom and Utility SAP customers. In company DigitalRoute, there is an equivalent product name called Mediation Zone, which is using a different product release version. Here is the product version mapping relationship between those two product names:
From SAP SE From DigitalRoute AB MZ version SAP Convergent Mediation 3.0 (out of maintenance) Mediation Zone 7.0 SAP Convergent Mediation 3.1 (out of maintenance) Mediation Zone 7.1 SAP Convergent Mediation 3.2 (out of maintenance) Mediation Zone 7.2 SAP Convergent Mediation 3.3 (out of maintenance) Mediation Zone 7.3 SAP Convergent Mediation 4.0 Mediation Zone 8.0 SAP Convergent Mediation 4.1 Mediation Zone 8.1 8.1.3.4 SAP Convergent Mediation 4.2 Mediation Zone 8.2 8.1.6.0 SAP Convergent Mediation 4.3 Mediation Zone 8.3 8.1.7.1 SAP Convergent Mediation 4.3.1 Mediation Zone 8.3.1 8.1.7.6
SAP GPS (Global Partner Support) supportability lab has worked with DigitalRoute development team on the Application Operation topic by utilizing SAP Solution Manager as the support infrastructure since 2015. The minimum production version supported is SAP Convergent Mediation 3.0 (Mediation Zone 7.0). For the old product versions, there is no plan to offer the Solution Manager integrations.
This document will introduce the Application Operation integration setup and configuration for SAP Convergent Mediation 4.0 in SAP Solution Manager 7.2. The old SAP Convergent Mediation versions (3.0 – 3.3) could use the similar procedures with minor adaptions, however it is recommended to upgrade to the new 4.0 version as version 3.3 will reach the end of the lifecycle in year 2020.
To solve the conflicts between Introscope Java bytecode agent instrumentation on Apache CXF framework (referenced in the Webservice related features) and Apache Flex OSGI container bootstrapper classloader restriction, it is highly recommend upgrading the system to 4.0 SP06 or above.
To support the Root Cause Analysis integration with SAP Solution Manager, a customized SAP supportability package is released and could be downloaded from SAP note 2718475. The detail instructions for that package is addressed in section 4.1.
SAP Solution Manager Requirement
To setup the Application Operation integration for SAP Convergent Mediation, the SAP Solution Manager System has to fulfil the following minimum requirements:
Component Release SAP Solution Manager 7.2 SP05 or above CA Introscope 10.5 or above Others SAPJVM installed by SAP Diagnostic Agent should not be in patch range 72 – 78 (SAP Note 2142276)
3 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION
Related SAP Notes
Please review the following SAP Notes and apply them into the SAP Solution Manager System if they are applicable.
SAP Note Number 2718475 2198172 2021921 2402129 2403541 2556432
To download the latest patch of SAPJVM 6.1, please refer to SAP Note 1442124 to get the download URL from SAP service market place. SAP note 1774669 introduces the correction approach for the SAPJVM patching process, however with SAP Diagnostic Agent, it is still needed to manually copy the complete sapjvm_6 folder from the SYS folder to exe folder.
PURPOSE
The purpose of this document is to provide the instructions for SAP customers who are using SAP Convergent Mediation products in their solution landscape and want to setup the Application Operation functions with SAP Solution Manager, e.g. Root Cause Analysis, System Monitoring …
In this document, it describes the integration setup procedures step by step within SAP Solution Manager 7.2 System and SAP Convergent Mediation 4.0 system. The target audience shall be familiar with the SAP Solution Manager System 7.2 administration as well as SAP Convergent Mediation 4.0 administration.
LANDSCAPE INTEGRATION
Understand SAP CM Landscape Modeling
SAP Convergent Mediation system has a cluster runtime architecture with 3 layers as illustrated in the below diagram. For the cluster wide, each SAP CM system is a cluster running cross more than one hosts. The 2 nd layer is so called container, which is composed of a group of JAVA Pico processes on each host. The 3 rd layer is the Pico process, which takes different function roles and serves for actual request loads. The container layer is a logical concept during the runtime, however, the SAP CM software installation on each host is container based, like SAP ABAP application server installation.
Each container contains at least one Pico process of the following types: - Platform: The platform Pico process is the central component in the cluster, which connects a central storage repository and coordinates events communication with other Pico processes. The platform Pico process is also used for the design time, where the SAP CM workflow objects are developed, configured and activated, then distributed to the registered Execution Context Pico processes. There is only 1
4 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION platform Pico process cluster wide (besides HA topic discussion). The container which has the platform Pico process is so called Control Zone. - Execution Context / Execution Context Standalone: The EC/ECSA Pico process is the working container for the actual runtime loads, where the designed workflows, batches, tasks are executed. - Service Context: The SC Pico process is used for serving certain entrance points of communication protocols. The container which has the EC/ECSA or SC Pico processes is so called Execution Zone.
The platform Pico process has a few options for the repository storage, using the Apache Derby in-process database, Oracle, PostgreSQL, SAP HANA RDBMS externally.
Diagram-1
SAP Convergent Mediation system is modelled as Unspecific Cluster System type in SAP landscape directory. The whole group of Pico processes are in one Technical System and each single Pico process is mapped as one Technical Instance. The container logical layer is skipped in the modelling due to the compatibility reasons.
Installation Best Practice and Naming Conventions
Here are some highly recommended suggestions when performing the SAP CM system installation:
a) Installation path: It is recommended to install the SAP CM software at path /usr/sap/
b) User group and permission: After the SAP Diagnostic Agent installation on SAP CM hosts, the OS group “sapsys” will be created. Please assign that group to SAP CM OS user “mzadmin”, e.g. using the following command:
$ usermod -a -G sapsys mzadmin
c) Configure the library path environment variable for SAP CM OS user “mzadmin” in its shell profile, e.g. in bash shell, the ~/.bash_profile will have the following highlighted item (if the OS is not Linux or the shell is not bash, the environment name and environment variable setting command might be a bit different):
export MZ_HOME=/usr/sap/MZ8 export JAVA_HOME=/usr/java/jdk1.8.0_111
5 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION export PATH=$JAVA_HOME/bin:$MZ_HOME/bin:$PATH
export LD_LIBRARY_PATH=/usr/sap/hostctrl/exe:$LD_LIBRARY_PATH
d) Container name: Although the container layer is skipped in the landscape modeling, it is recommended to provide a meaningful container name in the install.xml. A general naming rule will be using czXX for the control zone and ezXX for the execution zone. (XX would be the any numbers used to distinguish the hosts) e) Pico Process Name: It is highly recommended to use a customized script under {MZ_HOME}/scripts/str- templates after the SAP CM software installation on each host so that the Pico process name could be customized instead of using the default ones. The Pico process name will be used as the technical instance name in SAP SLD/LMDB and all Solution Manager applications. It is important to give it a meaningful name, which could make it easy for SAP CM system administrator to isolate the problem location, e.g. when receiving a system monitoring alert. A typical conversion would be
f) It is highly recommended to configure one Execution Context Pico process running on the same host of the platform for supportability integration purpose. This EC Pico process will host the CCDB workflow.
g) After the SAP CM system installation, please adjust the permission towards the log folder as using the following command (with mzadmin user OS account)
$ cd {MZ_HOME} $ chmod 755 etc $ chmod 755 log $ chmod -R +r log
SAP Diagnostic Agent Installation
SAP Diagnostic Agent must be installed on each host of the SAP Convergent Mediation system and connects to the Solution Manager System. SAP Host Agent will be installed as well when the Diagnostic Agent is installed. It is recommended installing the latest available SAP Diagnostic Agent release and applying the latest SAP Host Agent patch.
SLD Data Supplier Integration
To correctly register the SAP Convergent Mediation system to SAP SLD Server, it needs to use the generic SLD Data Supplier program to produce the corresponding SLD payload XML file and send it to the SLD Server via sldreg program. The detail registration operation is documented in SAP note 2021921. The SAP PPMS metadata files for SAP Convergent Mediation product versions are attached in SAP note 2718475.
6 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION Before starting the SLD registration procedure, please make sure that SAP Host Agent is installed together with the SAP Diagnostic Agent installation. Please also manually generate the slddest.cfg configuration file for sldreg under the SAP Host Agent executable folder.
Please use the following procedures step by step (All steps need to be performed on platform server only):
1) Verify SAP CM system SID and prepare the SAP CM system SLD parameters
Logon to platform server with mzadmin OS user, then switch to “{MZ_HOME}/sap-sldreg” folder
$ cd $MZ_HOME/sap-sldreg $ pwd /usr/sap/MZ8/sap-sldreg
Set the SAP CM system SLD parameters
$ mzsh topo set cell:default/val:user.sap.sld.cluster-install-path
2) Download prerequisite files and put them into target location
Download “insnamecollector.jar” from SAP note 2718475 and put it under “sap-sldreg” folder. Download “sldds4generic.jar” from SAP 2021921 and put it under “sap-sldreg” folder. Create a subfolder “/ppms/
3) Generate the initial cluster description file
Use the following commands to generate the initial version of “cluster-sld.conf”
$ mzsh mzadmin/
After executing this command line, a plain text file named “cluster-sld.conf” will be created under “sap- sldreg” folder. The file content is like the following:
$ more cluster-sld.conf ## ------# Cluster Server Domain Info ## ------
cluster-count=1 default-domain=mo.sap.corp ------cluster_0_sid=MZ8
7 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION cluster_0_name=MZ8 cluster_0_install-path=/usr/sap/MZ8 cluster_0_technology-type=SAPCM cluster_0_install-type=host cluster_0_host-count=2 cluster_0_host_0_ip=10.9.37.74 cluster_0_host_0_name=10.9.37.74 cluster_0_host_0_install-path=/usr/sap/MZ8 cluster_0_host_0_instance=ez01 cluster_0_host_0_useprefix=true cluster_0_host_1_ip=10.9.20.230 cluster_0_host_1_name=10.9.20.230 cluster_0_host_1_install-path=/usr/sap/MZ8 cluster_0_host_1_iscentral=true cluster_0_host_1_instance=cz01 cluster_0_host_1_useprefix=true
For the detail explanation of each property field, please refer to the document from SAP 2021921. There are 3 field values highlighted above need to be manually adapted and corrected, please refer to point 5.
4) Collect the Pico process distribution list (This step is not needed for CM 4.3 and above)
Use the following commands to get the Pico process list of each container
$ java -jar insnamecollector.jar
The output will show the following
INFO : Instance Name Collector for SAPCM 4.0 (Release: 1.0.0 - build:20181104) INFO : Comandline: java -jar insnamecollector.jar
INFO : Environment check succeeded, MZ_HOME=/usr/sap/MZ8 INFO : Start printing pico instance list for MZ containers <<<<<<<<
cz01=sc_sc3,sc_zk2,sc_psc1,ec_ec1,ecsa_ecsa1,sc_zk3,sc_sc2,sc_zk1,sc _sc1,platform_platform ez01=ec_ec2
INFO : End printing pico instance list for MZ containers <<<<<<<<
Copy the highlighted contents to a notepad editor.
5) Manually correct the cluster description file
Modify the “cluster-sld.conf” file for the following fields: a) “cluster_0_name” → Give it a more description name rather than just a
8 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION (Action b and c are not needed for CM 4.3 and above)
The modified “cluster-sld.conf” file would be like the following:
$ more cluster-sld.conf ## ------# Cluster Server Domain Info ## ------
cluster-count=1 default-domain=mo.sap.corp ------cluster_0_sid=MZ8 cluster_0_name=MZ8 Test System cluster_0_install-path=/usr/sap/MZ8 cluster_0_technology-type=SAPCM cluster_0_install-type=host cluster_0_host-count=2 cluster_0_host_0_ip=10.9.37.74 cluster_0_host_0_name=mo-0c158317c cluster_0_host_0_install-path=/usr/sap/MZ8 cluster_0_host_0_instance=ec_ec2 cluster_0_host_0_useprefix=true cluster_0_host_1_ip=10.9.20.230 cluster_0_host_1_name=mo-2fc9d1504 cluster_0_host_1_install-path=/usr/sap/MZ8 cluster_0_host_1_iscentral=true cluster_0_host_1_instance=sc_sc3,sc_zk2,sc_psc1,ec_ec1,ecsa_ecsa1,sc _zk3,sc_sc2,sc_zk1,sc_sc1,platform_platform cluster_0_host_1_useprefix=true
6) Run the generic SLD data supplier
Use the following commands to generate the SLD payload file and send it to SLD server via sldreg
$ java -jar sldds4generic.jar
OR (if Outside Discovery is enabled at the SAP Host Agent)
$ java -jar sldds4generic.jar odreg=true
The output will show the following if it runs successfully
INFO : Generic SLD Data Supplier (Release: 2.1.0 - build:20190424) INFO : mode = cluster INFO : infile = /usr/sap/MZ8/sap-sldreg/cluster-sld.conf INFO : outdir = /usr/sap/MZ8/sap-sldreg
9 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION INFO : testmode = false INFO : odreg = false INFO : Start SLD payload XML generation for unspecific cluster systems... Configuration file loaded at: /usr/sap/MZ8/sap-sldreg/cluster- sld.conf INFO: Check SAP_metadata.xml at - (/usr/sap/MZ8/sap- sldreg/ppms/MZ8/SAP_metadata.xml) INFO: Check SAP_metadata.xml at - (/usr/sap/MZ8/sap- sldreg/ppms/MZ8/SAP_metadata.xml), loading result is: true INFO: Succeeded loading the SCV assignment file scv-assignment.conf in the directory! INFO: Check scv_assignment.conf at - (/usr/sap/MZ8/sap- sldreg/ppms/MZ8/scv-assignment.conf), loading result is: true INFO : Use sldreg command for the SLD payload file registration INFO : Start checking sldreg runtime... INFO : Successfully checking sldreg runtime from SAP host agent, ready to send. INFO : sldreg executable file located at - /usr/sap/hostctrl/exe/sldreg INFO : sldreg configuration file located at - /usr/sap/hostctrl/exe/slddest.cfg INFO : Complete SLD payload XML generation, result payload file at:/usr/sap/MZ8/sap-sldreg/CLUSTER_PAYLOAD_0.xml INFO : Trigger SLD payload sending via commandline java.lang.ProcessBuilder@5acf9800 INFO : ------SLGREG sending start------(0). Command line used to start sldreg: /usr/sap/hostctrl/exe/sldreg -file /usr/sap/MZ8/sap- sldreg/CLUSTER_PAYLOAD_0.xml -connectfile /usr/sap/hostctrl/exe/slddest.cfg Profile is not explicitly specified on cmd line using pf=
10 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION Thu May 02 19:35:48 2019 HTTP trace: false Thu May 02 19:35:48 2019 Data trace: false Thu May 02 19:35:48 2019 ****************************** Thu May 02 19:35:48 2019 *** Start SLD Registration *** Thu May 02 19:35:48 2019 ****************************** Timeout '-1' means infinite. But '0' must be on cmd line. HTTP open timeout = 420 sec HTTP send timeout = 420 sec HTTP response timeout = 420 sec Thu May 02 19:35:48 2019 Used URL: http://mo- b8a81ce50.mo.sap.corp:50200/sld/ds Thu May 02 19:35:48 2019 Proxy is not configured. Thu May 02 19:35:48 2019 HTTP open status: true - NI RC=0 Thu May 02 19:35:48 2019 HTTP send successful. Thu May 02 19:35:48 2019 Return code: 200 --- Return message: (OK) Thu May 02 19:35:48 2019 HTTP response: HTTP status code: 200 Thu May 02 19:35:48 2019 Data send status: true INFO : sldreg process exit value: 0 INFO : ------SLGREG sending complete------(0).
If the SLD registration result is not successful, please check the following items: a) Check whether all actions are performed with the “mzadmin” OS user on the platform server host. b) Check whether the library path environment variable is correctly configured in the shell. See chapter 3.2, point c. c) Check whether “slddest.cfg” is correctly maintained.
In the future versions, the partner might improve this procedure further by moving more and more steps into the fully automatically tasks. The SLD data supplier integration procedure documented here is subject to change with new SAP CM version.
SUPPORTABILITY CONTENT PREPRATION
SAP Supportability Package for SAP CM
SAP Supportability Package is developed by SAP GPS team and used to integrate SAP Convergent Mediation system configuration data and logging data to SAP Solution Manager. The package is attached attached in SAP note 2718475.
Supportability Package Content
SAP Supportability Package contains the following elements:
11 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION
➢ Folder SAPSupport: folder used to store the CM elements created by SAP. ➢ APL code APL_APPL_ExceptionLogging: APL function code sample shows how to create SAP compatible application logging entry. The customer could copy the code sample to their own folder to avoid the project release dependency. ➢ EventNotifier ApplicationExceptionToFile: event notifier configuration for SAP compatible application logging entries. It defines the output filename and location of the application logging file, the output format of the logging entry and the selected event type used for the application logging. When the customer uses the provided application logging ULTRA class and APL sample code, each event will be re-directed to the application logging file during the runtime. ➢ External Reference sapcm_keystore: this element references to a local property where stores the credential of the interactive command via mzsh. It is used in the CCDB workflow to fetch the SAP CM workflow objects configuration information. The default file location is /usr/sap/ccms/sapcm_keystore. ➢ EventNotifier SystemLogToFile: a customized event notifier configuration to duplicate the SAP CM SystemLog (stored in the central repository database by default) content to the local logging file in SAP compatible logging format. The filtering criteria is enabled to limit the events which have the priority like Warning, Error and Disaster. ➢ ULTRA format ULFL_APPL_ExceptionLog_DataModel: ULTRA data model for the SAP compatible application logging entry. It defines the following logging entry fields: o location – indicate where the application exception happens. o exceptionId – customer defined exception identify, used for the remedy plan definition. o bizContext – context information when the exception happens, e.g. user transaction account. o dsrGUID – request UUID or batch UUID, used for the E2E exception correlation. o payloadRef – reference information for the payload storage, the payload information will be persisted to a local file on demand and later used for the root cause analysis. o message – exception message details.
➢ Real-time Workflow WFL_SAPCM_QueryConfig: this workflow object implements the CCDB web service interface and feed SAP Solution Manager the configuration data for SAP CM system library package and workflow objects. ➢ Web Service Profile WS_SAP_CCDB: this is web service profile object generated from the imported CCDB WSDL. It is used by the workflow object WFL_SAPCM_QueryConfig for web service input/output data marshaling.
In SAP note 2718475, there is a SAP sample package contains a simple workflow with WebService agents showing the coding reference for the customized system exception logging and application exception logging.
Supportability Package Import Procedure
12 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION SAP Supportability Package is released in a zip package format with filename pattern “SAPSupport_
1. Logon to Mediation Zone administration client UI using administrator user account, e.g. mzadmin
Choose “Tools” dropdown menu icon and select “System Importer” menu item.
2. Click “Browse…” button to pop up the file selection dialog window, navigate to the folder where SAPSupport package is stored and choose the package zip file.
3. Click “OK” button for the export message confirmation.
13 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION 4. Review the content catalog in the package file and click “Import” button to start the import process.
Confirm the pop-up import success message.
Post Import Customizing Actions
After successful importing the support package, there are some configurations need to be customized towards the managed SAP CM system:
➢ Maintain the local properties file for external reference resource
The default setting of the local properties file is “/usr/sap/ccms/sapcm_keystore”, if the customer prefers using a different location, the properties file setting value needs to be re-aligned. The properties file has to maintain two parameters: MZUSER and MZPASSWORD. This credential is used for the CCDB Web Service to issue the interactive command line for the workflow objects listing.
As the password is stored without encryption, this local property file has to be assigned with 600 permission, which means only SAP Convergent Mediation system OS user is allowed to read and write that properties file.
➢ Customizing the runtime execution context container configuration for CCDB workflow
14 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION
After the package importing, the CCDB workflow “WFL_SAPCM_QueryConfig” will keep the last runtime Execution Container configuration, which might not be valid in the managed system. The SAP CM system administrator has to re-assign the Execution Context Container from the system, normally we choose any of the EC/ECSA container running on the same host of the Platform container.
Please do not forget to activate that CCDB workflow after the configuration change. The default CCDB Web Service endpoint will be accessed at http://
➢ Activate the Event Notifier objects
After the package importing, all Event Notifier objects are in the de-activate status, please check on that “Event Notifier Enabled” checkbox and adjust that logging file rotation size to 10Mb. Keep the Directory and Filename setting untouched.
15 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION
The system exception logging should be activated always to duplicate the filtered system log error events to the local logging file. The content in that logging file will be integrated with SAP Solution Manager Exception Management application.
The application exception logging enabling is up to the customer.
➢ Enable the Application Exception logging (optional)
As introduced in the section 4.1.1, SAP provides the application exception ULTRA format data model and the APL code reference on how to trigger the application exception event in the APL programing. The customer has 2 options to apply that application exception integration:
a) The customer could use the model and APL code from the SAPSupport folder directly. b) The customer could copy the model and APL code from the SAPSupport folder to their own project folders. By doing so, they remove the dependency between their project folders to SAPSupport folder. But they need to re-configure the ApplicationExceptionToFile Event Notifier object and re-select the event type from customer project folder.
Similar as the SystemLogToFile Event Notifier object, ApplicationExceptionToFile Event Notifier must be enabled, and the logging file rotation size should be adjusted to 10Mb.
RCA Template for SAP CM
SAP RCA template is the XML format configuration file used to describe the important parameters for the SAP software installed in the technical system and to specify the important configuration data and logging files. Those templates are released in SAP Solution Manager as the RCA repository contents and used during the managed system setup procedure.
The latest RCA template for SAP Convergent Mediation 4.0 is released with SAP Solution Manager 7.2 SP09. For the old SP between SP05 and SP08, the customer could download that from the following SAP notes and manually upload the RCA template to SAP Solution Manager System in the “SAP Solution Manager Administration”.
SAP Solution Manager Release SAP Notes for RCA template download 7.2 SP05 2718475
16 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION 7.2 SP06 2677925 7.2 SP07 2677887 7.2 SP08
Download SAP CM RCA template file “SAPCM_RCA_Template_
1. Unzip the file to the local directory, the zip file should contain 6 XML files.
Software component version template files will be used as the baseline and newer versions will reuse those templates.
2. Logon to SAP Solution Manager as administrator user (SOLMAN_ADMIN). From the Fiori launchpad, choose the “SAP Solution Manager Administration” tile group, then click the “Template – Root Cause Analysis”.
If that tile is not visible in the group, search it and add it to the group using the launchpad personalization. 3. In the “Store” tab, browser each XML file from the extracted zip package and click “Custom Upload” button to import them into the repository content one by one.
4. Click “Show custom only” button below. The result list should display those 6 uploaded CV template contents.
17 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION Logging Format for SAP CM
In order to have SAP Diagnostic Agent successfully parse the system exception log and application exception log produced from those event notifiers, the customized logging format configuration file must be uploaded to the Solution Manager 7.2 system.
Download logging format configuration file “custom_GLS_config.xml” from the above SAP note 2718475 and use the following steps to upload them to SAP Solution Manager Agent Administration:
1. Logon to SAP Solution Manager as administrator user (SOLMAN_ADMIN). From the Fiori launchpad, choose the “SAP Solution Manager Administration” tile group, then click the “Agents Administration”.
2. Choose “Agent Admin” button -> “All Agents”.
3. In the “Application Configuration” tab, locate the application resource file “custom_GLS_config.xml” from the “logviewer” application, then upload the new configuration file from SAP note. The new resource content will be pushed to all agents.
If the resource file already contains the customized content for other products, then the content must be merged before the uploading.
MANAGED SYSTEM SETUP AND CUSTOMIZING
18 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION Managed System Setup
Before running the managed system setup, it is necessary to verify the SAP Convergent Mediation system in the Solution Manager Landscape Management to make sure all software components are correctly assigned, and the diagnostic relevant flag is set.
Verify the technical instance list and see whether all Execution Context containers are correctly named:
19 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION
Please use the following steps for the SAP Convergent Mediation System setup in SAP Solution Manager:
1. Logon to SAP Solution Manager as administrator user (SOLMAN_ADMIN), From the Fiori launchpad, choose the “SAP Solution Manager Configuration” tile group, then click the “Configuration – All Scenarios”.
Select the managed system SID of SAP CM system and choose “Configure System” > “Min. Configuration Without Service Delivery/EWA” 2. In the managed system setup guided procedure – step 1, check the diagnostic relevant assignment status.
The product system assignment and logical system assignment could be skipped for the moment.
3. In the step 2, click “Execute All” button in the edit mode.
20 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION 4. In the step 3, select the Introscope Enterpise Manager where the bytecode agent will connect to.
In the managed system load balancer information, provide the Platform Pico running host and the http port of the Platform container.
5. In the step 4, the landscape parameters could be customized for each technical instance. For the Platform Pico instance, additional CCDB Web Service parameters need to be provided, based on the settings customized in section 4.1.3
6. In the step 5, click “Execute All” button in the edit mode for automatic activities.
For manual activities, they could be skipped or manually set to “performed”.
7. In the step 6, click “Execute All” button in the edit mode for automatic activities.
21 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION 8. The step 7 is a summary step to display the previous steps’ execution status.
After performing the managed system setup for SAP Convergent Mediation system, the corresponding extractors will be activated and CCDB stores will be initialized. The managed SAPCM system is ready for the Root Cause Analysis applications and Technical Monitoring applications.
Customize Monitoring Parameter for SAPCM Technical Instance
The SAP Convergent Mediation system is registered in SLD Server and LMDB using the following technology type values for its instances.
Pico process type Technology type value Platform Platform_AppServer Execution Context EC_AppServer Execution Context Standalone ECSA_AppServer Service Context SC_AppServer
In general, the platform and EC/ECSA Pico processes will have the OSGI HTTP container service. It is necessary to define a custom monitoring parameter named “Z_HTTPDPORT” for those technical instance roles.
The custom parameter configuration guide is documented in “Using Custom Parameters.pdf” attached in SAP note 2718475.
22 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION
After adding this custom parameter, each technical instance (platform and EC/ECSA) must maintain that parameter value with the port number of its HTTPD service.
INTROSCOPE INSTRUMENT ACTIVATION
SAP GPS supportability lab team developed the Introscope instrumentation contents for SAP Convergent Mediation system. The contents include the bytecode agent directive configuration (PBD), JMX metric customizing (Profile customizing), bytecode agent extension (metric name format extension) and the Introscope Management Module (Dashboards).
Please check the SAP Convergent Mediation system version and apply the SP patch to SP06 for version 4.0.
Introscope Bytecode Agent configuration
SAP Convergent Mediation 4.0 instrumentation content is released in Introscope 10.5 content patch. The Agent side deployment must be done manually in the following sequence:
23 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION ➢ Download Agent installation package file (zip file) from SAP support launchpad (http://support.sap.com → Support packages & Patches → S → SAP SOLUTION MANAGER → SAP SOLUTION MANAGER 7.2 → AGENTS FOR MANAGED SYSTEMS → WILY INTRO AGT MIN 10.5 J5) and extract it to the SAP CM system servers. The recommended directory is “/usr/sap/ccms/wily”.
$ unzip ISAGNTJ5STDxx_x_xxxxxx.ZIP-d /usr/sap/ccms/ $ chown -R
➢ Backup the IntroscopeAgent.profile under folder “/usr/sap/ccms/wily/core/config” to a different name and copy IntroscopeAgent_sapcm.profile to IntroscopeAgent.profile
$ cd /usr/sap/ccms/wily/core/config $ mv IntroscopeAgent.profie IntroscopeAgent.profile.original $ cp IntroscopeAgent_sapcm.profile IntroscopeAgent.profile
➢ Modify the IntroscopeAgent.profile using text editor to specify the correct Introscope Enterprise Manager server host and port. If MoM cluster is used, specify the host and port of the MoM node.
➢ Verify that sap_ext_partner.jar agent extension file is located under “/usr/sap/ccms/wily/core/ext” directory.
➢ Copy WebAppSupport.jar from /usr/sap/ccms/wily/common to /usr/sap/ccms/wily/core/ext adjust the file permission same as other files in the target folder. (VERY IMPORTANT step, otherwise the JMX metric data will be missing)
$ cd /usr/sap/ccms/wily/common $ cp WebAppSupport.jar /usr/sap/ccms/wily/core/ext $ cd /usr/sap/ccms/wily/core/ext $ chmod 755 WebAppSupport.jar
➢ Add Introscope bytecode enabling parameters into SAP CM system Pico process runtime configuration files. In SAP CM 4.0, the Pico process runtime configuration files are stored in a central location and using the JSON style format.
24 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION $ cd $MZ_HOME/common/config/cell/default/master/containers/
For each Pico process, insert the following parameters:
config { …… jvmargs { args=[ "-javaagent:/usr/sap/ccms/wily/Agent.jar" ] …… properties { …… com.wily.introscope.agent.agentName="
After changing the configuration files, do remember to activate the change.
$ mzsh topo activate
➢ Restart SAP CM system using the normal sequence. The SC/EC/ECSA processes must be shut down before the Platform container process and opposite in the startup sequence.
➢ Logon to Introscope Enterprise server using workstation client or WebView client to make sure that each container process is reporting the metrics to the expected agent node name and the GC/JMX data are correctly populated.
With new SAP CM version, new Agent types might be introduced. The Introscope Agent instrumentation content might not be update to date to include those types. If there are missing instrumentations in SAP customer solutions, please report an incident to component XX-PART-WILY for the content enhancement.
Introscope Management Module configuration
The management module of SAP Convergent Mediation is delivered in the “SAP_partner.jar” file in the content shipment. By default, this file is not enabled. The customer needs to manually copy it to the target directory.
➢ Download SAP Introscope Management Module content release from SAP support launchpad (http://support.sap.com → Support packages & Patches → S → SAP SOLUTION MANAGER → SAP SOLUTION MANAGER 7.2 → WILY INTROSCOPE → WILY INTRO EM MGMT MODULES10.5).
25 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION
➢ Extract the content zip file to the Introscope Enterprise Manager installation path, e.g. /usr/sap/ccms/apmintroscope
➢ Copy “SAP_partner.jar” file to the active module directory. The latest “SAP_partner.jar” file is also available in SAP note 2718475.
$ cd /usr/sap/ccms/introscope/ManagementModule $ cp SAP_partner.jar /usr/sap/ccms/apmintroscope/config/modules
➢ Restart Introscope Enterprise Manager Server to make the SAP Convergent Mediation content accessible.
The Introscope dashboard content for SAP Convergent Mediation is introduced in the “SAPCM_Introscope_Dashboard.pdf” attached in SAP note 2718475. With each SAP Convergent Mediation new product version release, the content will be enhanced in the future SAP Introscope Management Module releases.
ROOT CAUSE ANALYSIS INTEGRATION
SAP Convergent Mediation offers the following integrations with SAP Solution Manage Root Cause Analysis applications by providing its configuration data and logging data.
Change Reporting and CCDB Stores
In SAP CV template for SAP Convergent Mediation, a bunch of configuration stores are defined in SAP Solution Manager used to track the changes in the configuration data. After the managed system setup procedure for the SAP Convergent Mediation system, those CCDB stores are initialized and the configuration data content for those stores are retrieved daily via CCDB extractors.
The customer could check those configuration data and their change history in the Solution Manager “Change Reporting” tile with bellowing steps:
26 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION
1. Logon to SAP Solution Manager as operation user. From the Fiori launchpad, choose the “Root Cause Analysis” tile group, then click the “Change Reporting” tile.
If that tile is not visible in the group, search it and add it to the group using the launchpad personalization. 2. Filter out and select the SID for SAP CM system, choose “Change Reporting” (Either Start Embedded or Start New Window)
3. The left side panel will show all the configuration stores in the landscape hierarchy. The following stores are specific to the SAP CM system:
FS: Container startup parameter values FS: System software versions WS: System package versions WS: Workflow versions
FS means File System based store. WS means CCDB Web
Service interface-based store.
Other features are all generic in the Change Reporting application. The customer could query those configuration data in any time frame, as well as the changes. They could also compare the configuration store content from the different SAP Convergent Mediation systems or even the same system in different timestamps.
The system package version store and workflow version store are based on the CCDB Web Service Interface and the content is offered from the CCDB workflow object developed in SAP supportability package. See section 4.1.1. The configuration data for the workflow will be enhanced in the future product versions to include the changing commit message and the latest changed timestamp information.
Logviewer
The Logviewer application in the Root Cause Analysis workcenter is used to offer a central place to search/filter the application or system critical logging events in SAP Solution Manager System. The customer does not need to telnet to different servers and check the log file content.
27 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION SAP Convergent Mediation system by default does not offer an explicit logging mechanism for its system and application events. It is using a centralized observer pattern to dispatch the event from the distributed Execution Context container processes to the Platform container process and store them into the repository, either in- process Apache Derby database or external databases. Later the administrator could query the events from the admin website of the Platform container process.
It is a bit difficult to directly consume those events stored in the SAP CM repository DB, instead SAP uses two customized event notifiers to dump the events into local logging files with the compatible SAP logging format. After the managed system setup procedure, those 2 logging groups are created automatically and represent the exceptional events for the system log and application exception log.
Besides the system log and exception log configured from the events output persistence, all other container runtime log files will be displayed in the logviewer as plain text log contents.
The customer could check event (logging) content data in the Solution Manager “Root Cause Analysis” tile group with bellowing steps:
1. Logon to SAP Solution Manager, locate the “Root Cause Analysis” tile group.
Select “Log Viewer” tile.
2. Filter out and select the SID for SAP CM system, choose “Log Viewer”
A new Log Viewer browser window will popup.
3. On the top right, a list of available log files (groups) will be displayed. Besides those standard output/error log file for each container process,
below.
Other features are all generic in the Log Viewer application. The customer could query those logging data in any filter conditions. The logging file group has already considered the rotation factor and will query the old files for the historical event data.
Exception Management
28 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION SAP Solution Manager Exception Management is one step further than the Log Viewer application. In that Log Viewer application, the logging file content is parsed and converted into SAP compatible logging format and each log entry is displayed as a single row of the record.
The Exception Management application could place further content level filtering criteria on the logging file group and display each logging entry as the atomic exception. The major benefit of the Exception Management is to include the atomic exceptions from different technical systems in the solution landscape and correlate those exceptions using business context correlation rules. Another benefit of the Exception Management application is to provide the short period exception data storage for the System Monitoring.
The configuration settings for the Exception Management application is not included in the managed system setup procedure. The customer needs to perform the following steps to enabling the integration for SAP Convergent Mediation system.
1. Logon to SAP Solution Manager as administrator user (SOLMAN_ADMIN), launch “SOLMAN_SETUP” t-code.
Navigate to “Application Operations” → “Exception Management”. Complete the general configurations in step 1 and step 2.
In step 3 – “Define Scope”, select the SAP CM system and click “Configure” button.
2. Click “Add Log Store” > “Standard”.
3. In the popup window, select that “systemlog.log* from Unspecified Instance” item and click OK button.
29 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION 4. In the next popup window, select the “Severity” field for filtering.
Customer could also add other fields for filtering like “Category”.
5. Specify the category name “SAPCM SystemLog”.
6. Adjust the filter conditions. Then click the “save” button.
The customer should create at least one Exception Category “SAPCM SystemLog” for the log store “systemlog.log*”. If the application exception logging is also enabled, the customer could create the second Exception Category “SAPCM ApplicationLog” for the log store “applicationlog.log*”, the filter condition could remain the same, e.g. Severity=Error or Severity=Fatal.
After activating the configurations, the filtered events will be displayed as the atomic exceptions in the Root Cause Analysis – Exception Monitoring.
Click the “Detail” button will launch the detail view for the selected atomic exception. In the future enhancement, it is planned to implement the E2E exception correlation for SAP Convergent Mediation exceptions with the other
30 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION technical system exceptions if they share the same message GUID, which means those exceptions are triggered from the same workflow execution in SAP CM system.
SYSTEM MONITORING INTEGRATION
The SAP Convergent Mediation System monitoring template is released as SAP Solution Manager monitoring content since STCONT SP26. The customer could update their monitoring content in the Solution Manager Configuration workcenter > Technical Monitoring > step 2.7 Update Content.
SAP Convergent Mediation System monitoring template contains one SAP monitoring template for the technical system and three templates for the technical instance, one for the Platform container process, one for the Execution Context container process and one for the Service Context container process.
The MAI content for SAP CM 4.0 is available since version 26.
When the customer starts configuring the system monitoring for the managed SAP Convergent Mediation system, they must firstly create the customer monitoring template inherited from the SAP template and adapt the template contents based on their managed system situation. The following factors might be considered:
➢ Whether the Application Exception logging is integrated with Solution Manager Exception Management? By default, in the system monitoring template, the metric defined for the application exception logging is disabled as it is up the customer to choose whether to use the SAP recommend interface to manage their application exceptions. If the customer is using SAP provided option, then the related metric and alert need to be activated in the template.
31 SAP CONVERGENT MEDIATION BY DIGITIALROUTE APPLICATION OPERATION INTEGRATION
➢ Select the corresponding agent types in the real-time workflow Agent performance metrics or batch workflow Agent performance metrics. As SAP does not know what kind of workflow scenario the customer implemented in their business solution, by default in the templates, only those non-static workflow agents like analysis agent and aggregation agent are enabled with the performance measurement metrics. For all other static workflow agents, no matter they are used in the real-time workflow or batch workflow, they are deactivated by default. The customer needs to activate them by choosing those agent types used in his SAP Convergent Mediation system.
➢ Threshold adjustment, all Agent performance metrics are delivered with the default threshold value setting, but the performance might vary from different load situation or different business solution designs. To avoid the alert flooding, the customer needs to refine those performance threshold settings according to their actual system behavior.
The complete alert and metric list is maintained in the “SAPCM_sysmon_content.xlsx” attached to SAP note 2718475. The content will be changed when the new SAP Convergent Mediation product versions are released, and the monitoring capabilities are enhanced in the future.
32
www.sap.com/contactsap
© 2021 SAP SE or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company.
The information contained herein may be changed without prior notice. Some software products marketed by SAP SE and its distributors contain proprietary softwar e components of other software vendors. National product specifications may vary.
These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.
In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation, and SAP SE’s or its affiliated companies’ strategy and possible future developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, and they should not be relied upon in making purchasing decisions.
SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trade marks of SAP SE (or an SAP affiliate company) in Germany and other countries. All other product and service names mentioned are the trademarks of their respective companies. See www.sap.com/trademark for additional trademark information and notices.