Upgrading a Multi Infrastructure EMC Controlcenter Environment from Version 6.0 to 6.1 UB7 on a Windows 2003 Cluster Kevin F Atkin

Total Page:16

File Type:pdf, Size:1020Kb

Upgrading a Multi Infrastructure EMC Controlcenter Environment from Version 6.0 to 6.1 UB7 on a Windows 2003 Cluster Kevin F Atkin Upgrading a multi Infrastructure EMC ControlCenter Environment from version 6.0 to 6.1 UB7 on a Windows 2003 Cluster Kevin F Atkin EMC Proven Professional Knowledge Sharing 2010 Kevin F Atkin SAN Support Engineer Thomson Reuters [email protected] Contents 1 OVERVIEW ..................................................................................................................................... 4 1.1 HOW TO USE THIS ARTICLE ............................................................................................................ 4 2 INSTALLATION PRE-REQUISITES .......................................................................................... 4 2.1 EMC CONTROLCENTER SERVER INFRASTRUCTURE BUILD .................................................... 4 2.2 HP DL380 SERVERS ...................................................................................................................... 6 2.2.1 Install Windows 2003 Enterprise Edition .......................................... 6 2.3 HP DL360 SERVERS ...................................................................................................................... 6 2.3.1 Install Windows 2003 Standard Edition ............................................ 6 2.3.2 Install HBA cards ............................................................................... 6 2.3.3 Install Powerpath version 5.3 ............................................................. 7 2.3.4 Zone and Mask Servers into the SAN................................................ 7 2.4 USER ACCOUNTS ........................................................................................................................... 8 2.4.1 Windows Cluster and User Accounts ................................................ 8 2.4.2 User Rights Assignment for the Clusteradmin Service Account ....... 9 2.4.3 ControlCenter Users........................................................................... 9 2.5 NETWORKING ............................................................................................................................... 10 2.5.1 Private LAN Network Card Configuration ...................................... 10 2.6 SERVICE LAN NETWORK CARD CONFIGURATION ...................................................................... 15 2.7 ADVANCED NETWORK CONFIGURATION ..................................................................................... 19 2.8 DNS NAME RESOLUTION ............................................................................................................ 20 2.9 INSTALL CONTROLCENTER IN A CLUSTERED ENVIRONMENT ..................................................... 20 2.9.1 Create the Cluster ............................................................................. 20 2.9.2 Create the Cluster Resources ........................................................... 24 2.9.3 Create Cluster Node Names ............................................................. 25 2.9.4 Create the EMC CONTROLCENTERRG Resource Group ........... 25 2.9.5 Create the shared physical disk resource ......................................... 26 2.9.6 Create the IP Address Resource ....................................................... 27 2.9.7 Create the NetworkName Resource ................................................. 29 2.9.8 Install Java on all Servers ................................................................ 32 2.9.9 Install MDAC 2.8 on the third HP DL 380 Server .......................... 32 2.9.10 Patch 4014 (Pre 6.1 StorageScope user fix) ..................................... 32 2.10 PREPARE HOSTS FOR CONTROLCENTER INSTALLATION ......................................................... 32 2.10.1 Upgrade Preparations ....................................................................... 34 2.10.2 Install ControlCenter onto the First Cluster Node ........................... 36 2.10.3 Upgrade Remaining Infrastructure Store Hosts ............................... 43 2.10.4 Upgrade StorageScope Host ............................................................ 44 2010 EMC Proven Professional Knowledge Sharing 2 2.10.5 Install the ControlCenter 6.1 Console .............................................. 48 2.10.6 ControlCenter 6.1 Post Upgrade Cluster checks.............................. 51 2.10.7 Edit EMC ControlCenter_server.ini File ......................................... 52 2.10.8 Edit the Master Agent file ctg.ini ..................................................... 53 2.10.9 Edit the EMC ControlCenter_store.ini file ...................................... 53 2.10.10 ControlCenter Patching .................................................................... 53 2.10.11 Patch 4667 (EMC CONTROLCENTER Repository) ..................... 53 2.10.12 Patch 4668 (StorageScope Repository) ........................................... 53 2.10.13 Solutions Enabler 7.1.0.0 ................................................................. 54 2.10.14 Install Solutions Enabler 7.1 MRLK (CC4762) on the Active node of the cluster. ........................................................................................................ 54 2.10.15 After EMC CONTROLCENTER 6.0 has Upgraded to EMC CONTROLCENTER 6.1 ..................................................................................... 54 2.10.16 Update Bundle 7 (Patch 4673) ......................................................... 57 2.10.17 Patch 4744 ........................................................................................ 58 2.10.18 Patch 4754 ........................................................................................ 58 2.10.19 Add local ORA_DBA Group to each Cluster Node ........................ 58 2.10.20 Post UB7 Cluster Preparation .......................................................... 59 2.10.21 Confirm Master Agents are at the same patch level ........................ 60 2.10.22 Agent Patching ................................................................................. 60 2.11 INSTALL CONTROLCENTER LICENCES .................................................................................... 61 2.12 CREATE CONTROLCENTER RESOURCES FOR THE CLUSTER SERVICE .................................... 62 2.12.1 Load StorageScope .......................................................................... 68 2.12.2 Install the WLA Archive Agent ....................................................... 69 2.12.3 Install the NAS Agent ...................................................................... 70 2.12.4 Add Gatekeepers to Symmetrix® Agent Host .................................. 71 Disclaimer: The views, processes, or methodologies published in this compilation are those of the authors. They do not necessarily reflect EMC Corporation’s views, processes, or methodologies. 2010 EMC Proven Professional Knowledge Sharing 3 1 Overview 1.1 How to use this Article I wrote this article based on my experience building an EMC ControlCenter® Cluster in the Thomson Reuters Data Center environments. Use this article in conjunction with the EMC Documentation available on PowerLink; I refer directly to EMC documentation in part, where I describe the installation process. This article s aimed at the Upgrade of EMC CONTROLCENTER 6.0 to 6.1 UB7 but includes the steps required to build the initial cluster etc. For existing EMC CONTROLCENTER 6.0 clustered installations, start from Section 2.9 to prepare the upgrade. 2 Installation Pre-Requisites 2.1 EMC CONTROLCENTER Server Infrastructure Build This article is based on a six server (two clustered) ControlCenter Infrastructure, but also applies to any number of hosts depending upon requirements. In this example, I have chosen the following server hardware. This procedure works equally well on HP Blade Servers too. • The EMC ControlCenter Server hardware will comprise 3 x HPDL380 Servers and 3 x HPDL360 Servers. • All servers that participate in EMC ControlCenter require the Windows 2003 Server. The HPDL380 Servers will require Windows 2003 Enterprise Edition for the purposes of clustering and to utilise the 8GB of Memory that they will be installed with. The HPDL360 Servers will be built using Windows 2003 Standard Edition. 2010 EMC Proven Professional Knowledge Sharing 4 • The ControlCenter Infrastructure will be installed as a 2 Node Cluster on HPDL380 Server hardware in a distributed configuration. The remaining HPDL380 Server will be used for Storage Scope and WLA (Work Load Analyser) reporting. The HPDL360 Servers will communicate with the EMC CONTROLCENTER Cluster to provide extra Console and Agent connectivity. • A reliable Windows Domain infrastructure and a Cluster Service Account that exists in the domain. • Three network connections in each clustered server (one for the Service LAN and two for Cluster Private LAN communication) with enough IP addresses to support all required resources: 1 x Server IP Address per node 2 x Cluster Private IP Addresses per Node 1 x Cluster IP Address 1 x Resource Group (Virtual Server) IP Address per Resource Group • Shared SAN storage that the clustered servers have access to and that are configured to EMC best practices. This will comprise a 1x 68GB LUN and a 500MB Lun for the Quorum disk. • Shared and Non shared 68GB LUNs will be assigned as follows: A) 1 x 68GB LUN assigned to the Cluster servers for Repository and the ControlCenter Server. This is will be assigned the drive letter R b) 1 x 68GB LUN assigned to the Storage Scope Repository on the third HPDL380 Server. This will be assigned the drive letter S C) 1x
Recommended publications
  • Bulk Data Migration Using Robocopy
    Bulk Data Migration Using Robocopy Created by: Nasuni Support Last update: 6/14/2011 Contents Overview...................................................................................................................................................................... 2 Using Robocopy to copy data to the Nasuni Filer....................................................................................... 2 2011 © Nasuni Corporation. All Rights Reserved Resizing the Cache and Snapshots Disks in Nasuni Filer v2.7 Robocopy and the Nasuni Filer cache ............................................................................................................. 2 Conclusion ................................................................................................................................................................... 3 Overview The information in this document applies to Nasuni Filer 2.x versions. It explains how to use Robocopy, a Microsoft Windows tool, to migrate data from a Windows file server to the Nasuni Filer. Robocopy has been bundled with Windows operating systems since Vista and Server 2008. Using Robocopy to copy data to the Nasuni Filer Robocopy is a Windows command line tool. To migrate data to the Nasuni Filer using Robocopy follow the steps below. 1. Identify the data set you want to migrate to the Nasuni Filer 2. Create/locate a share on the Nasuni Filer to write your data set to 3. Run cmd.exe 4. Run the command below from the source server’s command line. robocopy (path to source data) (path to Filer share)
    [Show full text]
  • Preparation of Papers in Two-Column Format
    Protecting Media Production Companies against Ransomware Rob Gonsalves Avid Technology, Inc. Burlington Massachusetts [email protected] Abstract - Ransomware, a type of malicious software designed to block access to digital assets until a sum of money is paid, represents a growing threat for media production companies, as digital media assets are a valued target for hackers. To protect their media assets and mitigate the risk of these types of attacks, companies can implement a set of security policies, procedures and systems. Deploying and operating Disaster Recovery (DR) systems with specific safeguards against ransomware will help companies retrieve valuable files without having to pay FIGURE 1 - INITIAL RANSOMWARE ATTACK cyber-criminals. This paper will discuss the growing threat of ransomware to media companies and cover the key The initial attack often comes when a user inadvertently concepts for understanding and building security and DR downloads and installs malware from a website. After systems with specific safeguards against ransomware. installation, the ransomware quietly searches for and Using these techniques, critical business functions can encrypts files. Its goal is to stay below the radar until it can continue in the event of an attack. find and encrypt all of the files that could be of value to the user. By the time the company is presented with the RANSOMWARE malware’s message with the ransom demand, the damage has already been done [2]. Ransomware is a form of malware that encrypts data files and holds these files for ransom. After the initial infection, the malware begins encrypting files on local drives, shared storage, and potentially other computers on the network.
    [Show full text]
  • When Powerful SAS Meets Powershell
    PharmaSUG 2018 - Paper QT-06 ® TM When Powerful SAS Meets PowerShell Shunbing Zhao, Merck & Co., Inc., Rahway, NJ, USA Jeff Xia, Merck & Co., Inc., Rahway, NJ, USA Chao Su, Merck & Co., Inc., Rahway, NJ, USA ABSTRACT PowerShell is an MS Windows-based command shell for direct interaction with an operating system and its task automation. When combining the powerful SAS programming language and PowerShell commands/scripts, we can greatly improve our efficiency and accuracy by removing many trivial manual steps in our daily routine work as SAS programmers. This paper presents five applications we developed for process automation. 1) Automatically convert RTF files in a folder into PDF files, all files or a selection of them. Installation of Adobe Acrobat printer is not a requirement. 2) Search the specific text of interest in all files in a folder, the file format could be RTF or SAS Source code. It is very handy in situations like meeting urgent FDA requests when there is a need to search required information quickly. 3) Systematically back up all existing files including the ones in subfolders. 4) Update the attributes of a selection of files with ease, i.e., change all SAS code and their corresponding output including RTF tables and SAS log in a production environment to read only after database lock. 5) Remove hidden temporary files in a folder. It can clean up and limit confusion while delivering the whole output folder. Lastly, the SAS macros presented in this paper could be used as a starting point to develop many similar applications for process automation in analysis and reporting activities.
    [Show full text]
  • How to Perform File System Backup and Restore with Netwrix File Server Change Reporter Technical Article
    HOW TO PERFORM FILE SYSTEM BACKUP AND RESTORE WITH NETWRIX FILE SERVER CHANGE REPORTER TECHNICAL ARTICLE Product Version: 3.3 April/2012 Copyright © 2012 NetWrix Corporation. All Rights Reserved. NetWrix File Server Change Reporter Technical Article Legal Notice The information in this publication is furnished for information use only, and does not constitute a commitment from NetWrix Corporation of any features or functions discussed. NetWrix Corporation assumes no responsibility or liability for the accuracy of the information presented, which is subject to change without notice. NetWrix is a registered trademark of NetWrix Corporation. The NetWrix logo and all other NetWrix product or service names and slogans are registered trademarks or trademarks of NetWrix Corporation. Active Directory is a trademark of Microsoft Corporation. All other trademarks and registered trademarks are property of their respective owners. Disclaimers This document may contain information regarding the use and installation of non-NetWrix products. Please note that this information is provided as a courtesy to assist you. While NetWrix tries to ensure that this information accurately reflects the information provided by the supplier, please refer to the materials provided with any non-NetWrix product and contact the supplier for confirmation. NetWrix Corporation assumes no responsibility or liability for incorrect or incomplete information provided about non-NetWrix products. © 2012 NetWrix Corporation. All rights reserved. www.netwrix.com Copyright © 2012 NetWrix Corporation. All Rights Reserved Page 2 of 11 Suggestions or comments about this document? www.netwrix.com/feedback NetWrix File Server Change Reporter Technical Article Table of Contents 1. INTRODUCTION ................................................................................ 4 1.1. Overview .............................................................................. 4 1.2.
    [Show full text]
  • Windows Tool Reference
    AppendixChapter A1 Windows Tool Reference Windows Management Tools This appendix lists sets of Windows management, maintenance, configuration, and monitor- ing tools that you may not be familiar with. Some are not automatically installed by Windows Setup but instead are hidden away in obscure folders on your Windows Setup DVD or CD- ROM. Others must be downloaded or purchased from Microsoft. They can be a great help in using, updating, and managing Windows. We’ll discuss the following tool kits: ■ Standard Tools—Our pick of handy programs installed by Windows Setup that we think are unappreciated and not well-enough known. ■ Support Tools—A set of useful command-line and GUI programs that can be installed from your Windows Setup DVD or CD-ROM. ■ Value-Added Tools—Several more sets of utilities hidden away on the Windows Setup CD-ROM. ■ Windows Ultimate Extras and PowerToys for XP—Accessories that can be downloaded for free from microsoft.com. The PowerToys include TweakUI, a program that lets you make adjustments to more Windows settings than you knew existed. ■ Resource Kits—A set of books published by Microsoft for some versions of Windows that includes a CD-ROM containing hundreds of utility programs. What you may not have known is that in some cases you can download the Resource Kit program toolkits with- out purchasing the books. ■ Subsystem for UNIX-Based Applications (SUA)—A package of network services and command-line tools that provide a nearly complete UNIX environment. It can be installed only on Windows Vista Ultimate and Enterprise, and Windows Server 2003.
    [Show full text]
  • A Records, 244–245, 279 -A Switch in Nbtstat, 190 in Netstat, 186 AAS Deployment Package, 710 .Aas Extension, 712 Abstract
    22_InsideWin_Index 13/3/03 9:50 AM Page 1269 Index A A records, 244–245, 279 ACEs (Access Control Entries) -a switch access masks in, 568–570 in Nbtstat, 190 command-line tools for, 572–576 in Netstat, 186 for cumulative permissions, 577 AAS deployment package, 710 for deny permissions, 578 .aas extension, 712 inheritance in, 579–580, 725–728 Abstract classes, 299–300 object ownership in, 572 Accelerated Graphics Port (AGP) adapters, 164 viewing and modifying, 570–571 Access Control Entries. See ACEs (Access ACKs in DHCP, 101–102 Control Entries) ACL Editor, 570, 723 Access control lists (ACLs) Advanced view in Active Directory security, 732–734 for inheritance, 578, 581 objects in, 339 for ownership, 572 in security descriptors, 559 for special permissions, 723–724 Access Control Settings window, 728 Edit view, 725–726 Access masks for permissions inheritance, 578 in ACEs, 568–570 blocking, 579 in DSOs, 733 settings for, 581 Access requests in Kerberos, 621 viewings, 582 Access rights. See also Permissions ACLs (access control lists) in Active Directory security in Active Directory security, 732–734 delegation, 729–732 objects in, 339 types of, 724–725 in security descriptors, 559 for group policies, 682 ACPI (Advanced Configuration and Power Access tokens Interface) contents of, 560–561 compatibility of, 23–28, 148–149 local, 559 kernel version for, 135 SIDs in, 559, 561, 581 for PnP,17, 147–149 ACCM (Asynchronous-Control- ACPIEnable option, 149 Character-Map), 1124 Activation Account domain upgrades, 496–498 in IA64, 130 BDC, 494–496 in installation, 49–50 PDC, 490–493 unattended setup scripts for, 95 Account lockout policies Active Directory, 238 in domain design, 429 bulk imports and exports in, 353–356 in password security, 593–594 DNS deployment in, 242–243 Account logons, auditing, 647 DNS integration in, 238–239 Account management, auditing, 511, 648 dynamic updates, 244–245 Accounts in domain migration.
    [Show full text]
  • Server-Side Data Migration Migrating Data from an Existing Volume to a Storsimple Volume
    Server-side Data Migration Migrating data from an existing volume to a StorSimple volume September 2, 2015 Copyright and Notices This document is provided "as-is". Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. Some examples depicted herein are provided for illustration only and are fictitious. No real association or connection is intended or should be inferred. This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy, use and modify this document for your internal, reference purposes. © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows Azure, StorSimple, Hyper-V, Internet Explorer, Silverlight, SQL Server, Windows, Windows PowerShell, and Windows Server are trademarks of the Microsoft group of companies. All other trademarks are property of their respective owners. The DiskBoss data management product is the property of Flexense Ltd. (see http://www.flexense.com/ and http://www.diskboss.com). Contents Copyright and Notices .......................................................................................................................................................................... 2 Contents .................................................................................................................................................................................................... 3 Introduction .............................................................................................................................................................................................
    [Show full text]
  • Are You Confident in Your Disaster Recovery Plan? Are You Confident in Your Disaster Recovery Plan?
    Are you confident in your Disaster Recovery Plan? Are you confident in your Disaster Recovery Plan? If your answer is “What Disaster Recovery Plan?” then this session is for you. All kidding aside, every year we have a small number of customers that lose data that could have been avoided if they had implemented an effective Disaster Recovery Plan. We encourage all System Admins to attend this session and learn how to be covered even when Murphy’s Law is in full effect. Agenda • Backup Plan on local Skyward server • Backup Plan on District Location • Disaster Recovery options • Test the Skyward backup file Local Skyward server Is there a backup process for the database? Yes, When the Role – Database Server installer was run an Online Backup task was setup in Task Scheduler – Windows. Local Skyward server What is the process that runs the backup? • Windows – dbscan.bat Local Skyward server How often does the backup process run? • Be default this process runs at 10PM nightly. Local Skyward server Where do the files get backed up? • \skyward\backup folder if "%PROPATH%" == "" set PROPATH=C:\Skyward set BackupDir=%PROPATH%\backup Local Skyward server What else is in the ?:\skyward\backup folder? copy %DLC%\properties\*.properties %BackupDir% copy %DLC%\startup.pf %BackupDir% copy %ConfigDir%\*.* %BackupDir%\config\ copy %CustRptDir%\*.* %BackupDir%\custreports\ • The OpenEdge parameter from the \dlc\properties folder. • The OpenEdge startup.pf file • The \config folder where setup files and Monolith CAS folder is located • The \custreports folder where custom Crystal reports are located. Local Skyward server Can I rebuild a server with the contents of the backup folder? Yes, Linux/AIX customers contact Skyward and Windows customers could do it themselves.
    [Show full text]
  • A-Z List of Windows CMD Commands — Also Included CMD Commands Commands PDF PDF
    A-Z List Of Windows CMD Commands — Also Included CMD Com... https://techlog360.com/all-windows-cmd-commands/?preview_id=43... A-Z List Of Windows CMD Commands — Also Included CMD Commands Commands PDF PDF Sabarinath CMD Commands Function A addusers Used to add and list users in a CSV file admodcmd Used for bulk modifying contents in an active directory Address Resolution Protocol is used to map IP address to arp the hardware address assoc Used to change associations for file extensions associat One step file association at Run a command at a specific time atmadm Display connection info of the ATM adapter attrib Used to change file attributes B bcdboot Used to create and repair a system partition bcdedit Used to manage boot configuration data Used to manage the Background Intelligent Transfer bitsadmin Service bootcfg Used to edit boot configuration in Windows break Enable/Disable break capability (CTRL+C) in CMD C cacls Used to change permissions of files call Used one batch program to call another certreq Used to request a certificate from a certification authority certutil Manage Certification Authority files and services cd Used to change folder (directory) or go to a specific one change Used to change terminal services chcp Displays the number of active console code page chdir Same as cd command chkdsk Used to check and repair issues in the disk chkntfs Used to check the NTFS file system choice Accept user input (via keyboard) to a batch file cipher Used to encrypt/decrypt files and folder cleanmgr Used clean temp files and recycle
    [Show full text]
  • Latest Version of Microsoft Robocopy GUI Download Use Robocopy to Pre-Seed Files for DFS Replication
    latest version of Microsoft Robocopy GUI download Use Robocopy to pre-seed files for DFS Replication. This topic explains how to use the command-line tool, Robocopy.exe , to pre-seed files when setting up replication for Distributed File System (DFS) Replication (also known as DFSR or DFS-R) in Windows Server. By pre-seeding files before you set up DFS Replication, add a new replication partner, or replace a server, you can speed up initial synchronization and enable cloning of the DFS Replication database in Windows Server 2012 R2. The Robocopy method is one of several pre-seeding methods; for an overview, see Step 1: pre-seed files for DFS Replication. The Robocopy (Robust File Copy) command-line utility is included with Windows Server. The utility provides extensive options that include copying security, backup API support, retry capabilities, and logging. Later versions include multi-threading and un-buffered I/O support. Robocopy does not copy exclusively locked files. If users tend to lock many files for long periods on your file servers, consider using a different pre-seeding method. pre-seeding does not require a perfect match between file lists on the source and destination servers, but the more files that do not exist when initial synchronization is performed for DFS Replication, the less effective pre-seeding is. To minimize lock conflicts, use Robocopy during non-peak hours for your organization. Always examine the Robocopy logs after pre-seeding to ensure that you understand which files were skipped because of exclusive locks. To use Robocopy to pre-seed files for DFS Replication, follow these steps: Prerequisites.
    [Show full text]
  • Pinpoint Labs Safecopy 2.0
    Pinpoint Labs SafeCopy 2.0 SafeCopy 2.0 enables the user to make forensically sound copies of files located on any drive–either local or networked. SafeCopy 2.0 features: ► High Speed Copying SafeCopy 2.0 has a unique bitstream copy engine that creates copies of files and directories that not only retain the file contents, but also retain the file timestamps and original metadata ensuring the proper chain of custody. This is all done 4-5 times faster than Robocopy. As the collections get larger, so does the time savings. ► Choice of Copy Engines On occasion, there will be the need for the selectable options available in Robocopy. SafeCopy 2.0 offers the choice for which engine to use when making copies of files and directories. Sniper ► Multiple Directory Selection SafeCopy 2.0 allows users to select multiple directories for copying. The easy-to- use interface allows for easy directory selection from within a familiar directory tree structure, or just drag and drop the files to be copied from the Windows Explorer interface. ► File Hashing When copying files,SafeCopy 2.0 retains the crucial metadata. To ensure the copies made with SafeCopy are identical to the originals, the user can calculate the MD5 hash values of the source files and destination files. ► Generation of Chain of Custody File SafeCopy 2.0 provides a chain of custody log file which proves that the copies made are forensically sound. This log file contains important information such as the date and time the copies were made and the MD5 hash values calculated in the source and destination files.
    [Show full text]
  • Acronis Files Connect Version 11.0.1
    Acronis Files Connect Version 11.0.1 USER GUIDE Revision: 6/8/2021 Table of contents 1 Introduction ....................................................................................................................... 4 2 QuickStart Guide ................................................................................................................ 5 2.1 Files Connect System Requirements ............................................................................................ 5 2.2 Installing Files Connect ................................................................................................................. 7 2.3 First Run / Installing Your License ................................................................................................ 8 2.4 Configuring Your First Shared Volume ......................................................................................... 8 2.5 Configuring Your First Shared Print Queue ................................................................................ 11 2.6 The Mac client ............................................................................................................................ 12 2.7 Connecting from the Mac Finder ............................................................................................... 18 2.8 The Optional Files Connect Zidget™ ........................................................................................... 19 2.9 Additional Resources .................................................................................................................
    [Show full text]