Autonomous Health Framework User’s Guide 21c F31833-01 November 2020 Autonomous Health Framework User’s Guide, 21c F31833-01 Copyright © 2016, 2020, Oracle and/or its affiliates. Primary Authors: Nirmal Kumar, Janet Stern Contributing Authors: Richard Strohm, Mark Bauer, Douglas Williams, Aparna Kamath, Subhash Chandra Contributors: Girdhari Ghantiyala, Gareth Chapman, Robert Caldwell, Vern Wagman, Mark Scardina, Ankita Khandelwal, Girish Adiga, Walter Battistella, Jesus Guillermo Munoz Nunez, Sahil Kumar, Daniel Semler, Carol Colrain This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable: U.S. GOVERNMENT END USERS: Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs) and Oracle computer documentation or other Oracle data delivered to or accessed by U.S. Government end users are "commercial computer software" or "commercial computer software documentation" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, reproduction, duplication, release, display, disclosure, modification, preparation of derivative works, and/or adaptation of i) Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs), ii) Oracle computer documentation and/or iii) other Oracle data, is subject to the rights and limitations specified in the license contained in the applicable contract. The terms governing the U.S. Government’s use of Oracle cloud services are defined by the applicable contract for such services. No other rights are granted to the U.S. Government. This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Inside are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Epyc, and the AMD logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle. Contents Preface Audience xvii Documentation Accessibility xvii Related Documentation xvii Conventions xviii 1 Introduction to Oracle Autonomous Health Framework 1.1 Oracle Autonomous Health Framework Problem and Solution Space 1-1 1.1.1 Availability Issues 1-1 1.1.2 Performance Issues 1-3 1.2 Components of Autonomous Health Framework 1-4 1.2.1 Introduction to Oracle Autonomous Health Framework Configuration Audit Tools 1-5 1.2.2 Introduction to Cluster Health Monitor 1-5 1.2.3 Introduction to Oracle Trace File Analyzer 1-6 1.2.4 Introduction to Oracle Cluster Health Advisor 1-6 1.2.5 Introduction to Memory Guard 1-7 1.2.6 Introduction to Hang Manager 1-8 1.2.7 Introduction to Oracle Database Quality of Service (QoS) Management 1-8 Part I Analyzing the Cluster Configuration 2 Analyzing Risks and Complying with Best Practices 2.1 Using Oracle ORAchk and Oracle EXAchk to Automatically Check for Risks and System Health 2-2 2.2 Email Notification and Health Check Report Overview 2-3 2.2.1 First Email Notification 2-4 2.2.2 What does the Compliance Check Report Contain? 2-4 2.2.3 Subsequent Email Notifications 2-6 2.3 Configuring Oracle ORAchk and Oracle EXAchk 2-6 2.3.1 Deciding Which User Should Run Oracle ORAchk and Oracle EXAchk 2-6 iii 2.3.2 Handling of Root Passwords 2-8 2.3.3 Configuring Email Notification System 2-9 2.4 Using Oracle ORAchk and Oracle EXAchk to Manually Generate Compliance Check Reports 2-13 2.4.1 Running Compliance Checks On-Demand 2-13 2.4.2 Running Compliance Checks in Silent Mode 2-15 2.4.3 Running On-Demand With or Without the Daemon 2-17 2.4.4 Generating a Diff Report 2-17 2.4.5 Sending Results by Email 2-18 2.5 Managing the Oracle ORAchk and Oracle EXAchk Daemons 2-18 2.5.1 Starting and Stopping the Daemon 2-18 2.5.2 Configuring the Daemon for Automatic Restart 2-19 2.5.3 Setting and Getting Options for the Daemon 2-20 2.5.3.1 AUTORUN_SCHEDULE 2-21 2.5.3.2 AUTORUN_FLAGS 2-22 2.5.3.3 NOTIFICATION_EMAIL 2-23 2.5.3.4 collection_retention 2-24 2.5.3.5 PASSWORD_CHECK_INTERVAL 2-24 2.5.3.6 Setting Multiple Option Profiles for the Daemon 2-25 2.5.3.7 Getting Existing Options for the Daemon 2-26 2.5.4 Querying the Status and Next Planned Daemon Run 2-30 2.6 Tracking Support Incidents 2-31 2.7 Tracking File Attribute Changes and Comparing Snapshots 2-32 2.7.1 Using the File Attribute Check With the Daemon 2-33 2.7.2 Taking File Attribute Snapshots 2-34 2.7.3 Including Directories to Check 2-34 2.7.4 Excluding Directories from Checks 2-35 2.7.5 Rechecking Changes 2-35 2.7.6 Designating a Snapshot As a Baseline 2-36 2.7.7 Restricting System Checks 2-37 2.7.8 Removing Snapshots 2-37 2.8 Collecting and Consuming Health Check Data 2-38 2.8.1 Selectively Capturing Users During Login 2-38 2.8.2 Bulk Mapping Systems to Business Units 2-40 2.8.3 Adjusting or Disabling Old Collections Purging 2-41 2.8.4 Uploading Collections Automatically 2-43 2.8.5 Viewing and Reattempting Failed Uploads 2-45 2.8.6 Authoring User-Defined Checks 2-46 2.8.7 Finding Which Checks Require Privileged Users 2-50 2.8.8 Creating or Editing Incidents Tickets 2-51 2.8.8.1 Creating Incident Tickets 2-52 iv 2.8.9 Viewing Clusterwide Linux Operating System Compliance Check (VMPScan) 2-52 2.9 Integrating Compliance Check Results with Other Tools 2-53 2.9.1 Integrating Compliance Check Results with Oracle Enterprise Manager 2-54 2.9.2 Integrating Compliance Check Results with Third-Party Tool 2-55 2.9.3 Integrating Compliance Check Results with Custom Application 2-57 2.10 Using Oracle ORAchk to Confirm System Readiness for Implementing Application Continuity 2-59 2.10.1 Overview of Application Continuity 2-59 2.10.2 Checks for Application Continuity 2-59 2.10.3 Application Continuity Protection Check 2-64 2.11 Configuring Oracle REST Data Services (ORDS) 2-64 2.11.1 Configuring REST Using the Included ORDS 2-65 2.11.2 Configuring REST Using an Existing ORDS Installation 2-65 2.12 Using Oracle Autonomous Health Framework Compliance Over REST 2-66 2.12.1 check 2-67 2.12.2 checktfafaileduploads 2-67 2.12.3 checktfaupload 2-68 2.12.4 download 2-68 2.12.5 getinfo 2-69 2.12.6 listcollections 2-70 2.12.7 gettfaupload 2-70 2.12.8 profile 2-71 2.12.9 showrepair 2-72 2.12.10 start_client 2-73 2.12.11 start_client 2-73 2.12.12 start_client 2-74 2.12.13 status 2-75 2.12.14 unsettfaupload 2-76 2.12.15 uploadtfafailed 2-76 2.12.16 version 2-77 2.13 Command-Line Options to Generate Password Protected Collection zip Files 2-77 2.14 Caching Discovery Data 2-79 2.15 Applying Patch Between Releases 2-80 2.16 Creating, Modifying, and Deleting User-Defined Profiles 2-80 2.17 Sanitizing Sensitive Information in the Diagnostic Collections 2-82 Sanitizing Sensitive Information in Oracle ORAchk or Oracle EXAchk Output 2-83 2.18 Troubleshooting Compliance Framework (Oracle ORAchk and Oracle EXAchk) 2-85 2.18.1 How to Troubleshoot Oracle ORAchk and Oracle EXAchk Issues 2-86 2.18.2 How to Capture Debug Output 2-87 v 2.18.3 Remote Login Problems 2-88 2.18.4 Permission Problems 2-90 2.18.5 Slow Performance, Skipped Checks, and Timeouts 2-91 3 Proactively Detecting and Diagnosing Performance Issues for Oracle RAC 3.1 Oracle Cluster Health Advisor Architecture 3-2 3.2 Monitoring the Oracle Real Application Clusters (Oracle RAC)