EVALUATION COPY Unauthorized Reproduction or Distribution Linux Prohibited Troubleshooting Student Workbook EVALUATION COPY Unauthorized Reproduction GL314 or LINUX Distribution TROUBLESHOOTING RHEL7 SLES12 Prohibited The contents of this course and all its modules and related materials, including handouts to audience members, are copyright ©2017 Guru Labs L.C. No part of this publication may be stored in a retrieval system, transmitted or reproduced in any way, including, but not limited to, photocopy, photograph, magnetic, electronic or other record, without the prior written permission of Guru Labs. This curriculum contains proprietary information which is for the exclusive use of customers of Guru Labs L.C., and is not to be shared with personnel other than those in attendance at this course. This instructional program, including all material provided herein, is supplied without any guarantees from Guru Labs L.C. Guru Labs L.C. assumes no liability for damages or legal action arising from the use or misuse of contents or details contained herein. Photocopying any part of this manual without prior written consent of Guru Labs L.C. is a violation of federal law. This manual should not appear to be a photocopy. If you believe that Guru Labs training materials are being photocopied without permission, please email [email protected] or call 1-801-298-5227. Guru Labs L.C. accepts no liability for any claims, demands, losses, damages, costs or expenses suffered or incurred howsoever arising from or in connection with the use of this courseware. All trademarks are the property of their respective owners. Version: GL314S-R7S12-H02 EVALUATION COPY Unauthorized Reproduction or Distribution Prohibited EVALUATION COPY Unauthorized Table of Contents Chapter 1 Recovery: Network Utilities 7 TROUBLESHOOTING METHODOLOGY 1 Lab Tasks 8 The Troubleshooting Mindset 2 1. Recovery Runlevels 9 Evaluating Possible Solutions 3 2. Recovering Damaged MBR 11 Identifying and Implementing Change 4 3. Recover from Deleted Critical Files 15 Define and Follow PoliciesReproduction 5 Working with Others 6 Chapter 4 Finding Documentation 7 TOPIC GROUP 1 1 Finding Help Online 9 Linux Boot Process 2 System Boot Method Overview 3 Chapter 2 systemd System and Service Manager 4 TROUBLESHOOTING TOOLS 1 Modifying systemd services 6 Common Troubleshooting Tools 2 Using systemd 7 RPM Queries 3 Booting Linux on PCs 9 RPM Verification or 4 Troubleshooting With GRUB 2 11 SRPM and spec Files 5 Boot Process Troubleshooting 13 Hardware Discovery Tools Distribution6 Troubleshooting: Linux and Init 14 Configuring New Hardware with hwinfo 7 Process Management 15 strace and ltrace 8 Process Management Tools 16 lsof and fuser 10 Troubleshooting Processes: top 17 ipcs and ipcrm 12 Filesystem Concepts 18 iostat, mpstat, and vmstat 13 Filesystem Troubleshooting 19 Using hdparm to Measure 15 Backup Concepts 21 Troubleshooting with the ip command 16 Backup Troubleshooting 22 Name Resolution 18 ProhibitedBackup Troubleshooting 23 ss/netstat and rpcinfo 20 Lab Tasks 24 nmap 21 1. Troubleshooting Problems: Topic Group 1 25 Netcat 22 tcpdump and wireshark 23 Chapter 5 Lab Tasks 27 TOPIC GROUP 2 1 1. Determining the System's Configuration 28 Networking Tools 2 2. Troubleshooting with rpm 43 Linux Network Interfaces 3 3. Process Related Tools 46 Networking Commands Review 5 4. Network Tools 54 NetworkManager 6 Networking Troubleshooting 8 Chapter 3 Networking Troubleshooting 9 RESCUE ENVIRONMENTS 1 Virtual Interfaces/IP Aliases 10 Diagnostic/Recovery 2 Network Teaming 11 Rescue Procedures 3 Xinetd Concepts 15 Recovery: mount & chroot 4 Xinetd Troubleshooting 16 Recovery Examples 6 TCP Wrappers Concepts 17 ii Unauthorized TCP Wrappers Concepts 18 1. Troubleshooting Problems: Topic Group 4 17 TCP Wrappers Troubleshooting 19 Netfilter/iptablesEVALUATION Concepts 20 Chapter 8 COPY Netfilter/iptables Troubleshooting 21 TOPIC GROUP 5 1 Lab Tasks 22 Kernel Modules 2 1. Troubleshooting Problems: Topic Group 2 23 Kernel Modules Troubleshooting 3 Logical Volume Management 4 Chapter 6 Reproduction Creating Logical Volumes 5 TOPIC GROUP 3 1 LVM Deployment Issues 6 X11 Concepts 2 VG Migration, PV Resizing & Troubeshooting 7 X11 Server Operation 3 Software RAID Overview 9 X11 Troubleshooting 4 RAID Troubleshooting 10 Rsyslog Concepts 6 Multipathing Overview 11 System Logging 7 SAN Multipathing 12 systemd Journal 9 Multipath Configuration 13 systemd Journal's journalctl 11 Multipathing Best Practices 15 Secure Logging with Journal's Log Sealing or 13 LDAP and OpenLDAP 17 Syslog Troubleshooting 15 Troubleshooting OpenLDAP 18 RPM Concepts Distribution16 NIS and NIS+ (YP) 20 RPM Troubleshooting 17 NIS Troubleshooting Aids 21 Common Unix Printing System (CUPS) 18 Lab Tasks 22 CUPS Troubleshooting 20 1. Troubleshooting Problems: Topic Group 5 23 CUPS Troubleshooting 21 at & cron 22 Chapter 9 at & cron Usage 23 TOPIC GROUP 6 1 at & cron Troubleshooting 24 DNS Concepts 2 Lab Tasks 25 ProhibitedDNS Troubleshooting 3 1. Troubleshooting Problems: Topic Group 3 26 DNS Troubleshooting 4 Apache Concepts 5 Chapter 7 Apache Troubleshooting 6 TOPIC GROUP 4 1 Apache Troubleshooting 7 Users and Groups 2 FTP Concepts 8 Users and Groups Troubleshooting 3 FTP Troubleshooting 9 PAM Concepts 4 Squid Concepts 10 PAM Troubleshooting 5 Squid Troubleshooting 11 Filesystem Quotas 6 Lab Tasks 12 Quotas Troubleshooting 7 1. Troubleshooting Problems: Topic Group 6 13 File Access Control Lists 8 FACL Troubleshooting 9 Chapter 10 SELinux Concepts 10 TOPIC GROUP 7 1 SELinux Troubleshooting 12 Samba Concepts 2 SELinux Troubleshooting Continued 14 Samba Troubleshooting 3 Lab Tasks 16 Postfix Concepts 4 iii Unauthorized Postfix Troubleshooting 6 Postfix Troubleshooting 8 IMAP & POP ConceptsEVALUATION9 COPY IMAP/POP Troubleshooting 10 MariaDB 11 MariaDB Troubleshooting 12 Lab Tasks 13 1. Troubleshooting Problems:Reproduction Topic Group 7 14 or Distribution Prohibited iv Unauthorized Typographic ConventionsEVALUATION COPY The fonts, layout, and typographic conventions of this book have been carefully chosen to increase readability. Please take a moment to familiarize yourself with them. A Warning and SolutionReproduction A common problem with computer training and reference materials is the confusion of the numbers "zero" and "one" with the letters "oh" and 0The number OThe letter "ell". To avoid this confusion, this book uses a fixed-width font that makes "zero". "oh". each letter and number distinct. Typefaces Used and Their Meanings The following typeface conventions have been followedor in this book: fixed-width normal ) Used to denote file names and directories. For example, the /etc/passwd file or /etc/sysconfig/Distributiondirectory. Also used for computer text, particularily command line output. 1The number lThe letter "one". "ell". fixed-width italic ) Indicates that a substitution is required. For example, the string stationX is commonly used to indicate that the student is expected to replace X with his or her own station number, such as station3. fixed-width bold ) Used to set apart commands. For example, theProhibited sed command. Also used to indicate input a user might type on the command line. For example, ssh -X station3. fixed-width bold italic ) Used when a substitution is required within a command or user input. For example, ssh -X stationX. fixed-width underlined ) Used to denote URLs. For example, http://www.gurulabs.com/. variable-width bold ) Used within labs to indicate a required student action that is not typed on the command line. Occasional variations from these conventions occur to increase clarity. This is most apparent in the labs where bold text is only used to indicate commands the student must enter or actions the student must perform. v Unauthorized Typographic ConventionsEVALUATION COPY Terms and Definitions Line Wrapping The following format is used to introduce and define a series of terms: Occasionally content that should be on a single line, such as command line input or URLs, must be broken across multiple lines in order to fit deprecate ) To indicate that something is considered obsolete, with on the page. When this is the case, a special symbol is used to indicate the intent of future removal.Reproduction to the reader what has happened. When copying the content, the line frob ) To manipulate or adjust, typically for fun, as opposed to tweak. breaks should not be included. For example, the following hypothetical grok ) To understand. Connotes intimate and exhaustive knowledge. PAM configuration should only take two actual lines: hork ) To break, generally beyond hope of repair. hosed ) A metaphor referring to a Cray that crashed after the password required /lib/security/pam_cracklib.so retry=3a disconnection of coolant hoses. Upon correction, users were assured type= minlen=12 dcredit=2 ucredit=2 lcredit=0 ocredit=2 the system was rehosed. password required /lib/security/pam_unix.so use_authtok mung (or munge) ) Mash Until No Good: to modify a file, often Representing File Edits irreversibly. troll ) To bait, or provoke, an argument, often targetedor towards the File edits are represented using a consistent layout similar to the unified newbie. Also used to refer to a person that regularly trolls. Distribution diff format. When a line should be added, it is shown in bold with a twiddle ) To make small, often aimless, changes. Similar to frob. plus sign to the left. When a line should be deleted, it is shown struck out with a minus sign to the left. When a line should be modified, it When discussing a command, this same format is also used to show and is shown twice. The old version of the line is shown struck out with a describe a list of common or important command options. For example, minus sign to the left. The new version of the line is shown below the the following ssh options: old version, bold and with a plus sign to the left. Unmodified lines are often included to provide context for the edit. For example, the following -X ) Enables X11 forwarding. In older versions of OpenSSH that do describes modification of an existing line and addition of a new line to not include -Y, this enables trusted X11 forwarding.