Re-Imaging an Exadata Storage Cell Using an Internal Or External Usb Drive
Total Page:16
File Type:pdf, Size:1020Kb
RE-IMAGING AN EXADATA STORAGE CELL USING AN INTERNAL OR EXTERNAL USB DRIVE www.hexaware.com Table of Contents Oracle Exadata Storage Server Software Rescue Procedure – A Brief Introduction 3 A Fast Forward experience to backup/restore/recovery 3 testing of your Future Exadata Environment Cell Node Re-Imaging - Ideal Scenarios 3 Pragmatic Scenarios where cell node re-imaging is not required 3 Recovering a Cell Node using Internal USB 4 Recovering Cell Node using External USB 7 Summary 7 1 Oracle Exadata Storage Server Software Rescue Procedure - A Brief Introduction A Fast Forward experience to backup/restore/recovery testing of your Future Exadata Environment • Have you experienced a disk failure and are deeply concerned about your operating system? • Has your system volume got corrupted due to the loss of disks? • Does your operating system have a corrupt file system? You can resolve these issues yourself using the Exadata Cell rescue functionality provided by Oracle on Exadata Storage Server Software CELLBOOT USB flash drive. The whole procedure is quite straightforward, but enough due diligence must be done to make the recovery a delightful experience. Oracle performs automatic backups of the operating system and cell software on each Exadata Storage Server without any Oracle DMA or operational process intervention. The critical files from the storage cells are backed up into the internal USB drive called CELLBOOT USB Flash Drive. The internal USB (/dev/sdm) maintains the most recent configuration and the updated copy of the OS and storage cell software. The internal USB serves as the default first boot device. The Master Boot Record (MBR) and GRand Unified Boot loader (GRUB) are loaded from the USB. The HDD (Hard Disk Drive) acts as a fallback maintaining its own copy of the above data. This paper describes the recovery process of a storage cell using the internal USB drive when both system disks are lost. This paper also outlines the cell node recovery procedure when both the system disks, and the internal USB drive are lost and no cell boot image is created on an external USB drive as a backup. Cell Node Re-Imaging - Ideal Scenarios The recommended scenarios to go for cell node re-imaging are: • Corruption of MD of a critical partition. • Corruption of an unrecoverable file system of a critical partition. • Accidental Inconsistent data on disk (due to rm-rf). Pragmatic Scenarios where cell node re-imaging is not required • When a non-critical partition is unrecoverable. • When file system corruption is minimal and file system correction check (fsck) is able to remediate. • When an inconsistent state of MD RAID can be manually remedied. CRITICAL PARTITIONS CRITICAL PARTITIONS • Active partitions are critical: /dev/md5 (mirrored to • /dev/md11 on /var/log/oracle stores cell logs and is /dev/md6) on /(root FS) and /dev/md7 (mirrored to therefore non-critical. This can be recovered by /dev/md8) on /opt/oracle) reformatting. • Internal usb (/dev/sdm1) itself is not critical and can be rebuilt, when booting the cell from disk. This normally takes place automatically while the cell is running the boot scripts. If not, manually run ./make_cellboot_usb.sh -force–verbose. • /dev/md4 on /boot – contents can be copied from a good cell whose image version is identical. 2 Recovering a Cell Node using Internal USB The following steps outline the process to be followed when both system disks (volumes) are lost. The following steps help invoke the storage cell rescue procedure. NOTE: It is strongly recommended to open a Service Request and engage Oracle Support before starting the Cell rescue procedure. 1. The Exadata storage server can be accessed using the ILOM console. After the cell is booted, the Oracle Exadata splash screen 2. From the boot options listed, the user should select: CELL_USB_BOOT_CELLBOOT_usb_in_rescue_mode. 3. After the node has been booted, two choices would be displayed to the user: (e)nter interactive diagnostic shell (r)einstall or try to recover corrupted system (r)einstall will delete, recreate and restore all partitions and system volumes from the data that is backed up to the CELLBOOT USB image Choose (r) to reinstall and confirm with 'y' to continue. 3 4. The rescue process would prompt the user for the “root rescue password”. This password is not the server’s root password. The default root rescue password is ‘‘sos1exadata”. 5. The system prompts the user whether to erase the data partition and data disks. The decision is passed based on the type of recovery that the user intends to perform. • When the user chooses to erase the data partition and data disks, the entire cell disk metadata will be removed from non-system disks. The user will have to reconfigure the storage server. • When user does not choose to erase the data partition and data disks, the entire cell disk metadata will be intact and the user can import the cell disks at a later stage. 6. The following action needs to be carried out when the system prompts for restart or provides options for entering shell at the end of first phase of the rescue: a. The user is required to enter the shell and not restart the system b. Log in using the rescue root password in the shell [Note: password is "sos1exadata"] c. The reboot command has to be run from the shell. The boot device selection menu can be accessed by pressing F8. F8 should be pressed as the cell restarts and before Oracle Exadata splash screen gets displayed. The user should select the RAID controller as the boot device. This enables the cell to boot from hard disks instead of CELLBOOT USB. 7. Once the rescue process is completed, the cell reboot will resume. In cases when the storage cell is powered off, the cell could be powered on by logging into the ILOM console. 8. After the cell boots, the user needs to validate the image information of the rescued cell with that of the other healthy cells. 9. After successful cell recovery, the cell must be configured. Since the data was preserved (by not erasing all data partitions), the cell disks can be imported a. The following command is used to import grid disks on all disks other than the ones that failed and were replaced during rescue procedure: CellCLI> IMPORT CELLDISK ALL FORCE Then confirm with ‘LIST CELLDISK’ command. b. The Cell Disks and Grid Disks should be re-created on the Physical Disks (System Disks) that were replaced during the rescue procedure. • Find out the Cell Disk names from the output of "list lun" and "list celldisk" commands. Example is shown below: CellCLI> list lun attributes name,deviceName,diskType where isSystemLun=TRUE 0_0 /dev/sdq HardDisk 0_1 /dev/sdr HardDisk CellCLI> list celldisk attributes name,deviceName,lun CD_00_celadm01 /dev/sdq 0_0 CD_01_celadm01 /dev/sdr 0_1 (………….. Output omitted for brevity) • Create the Cell Disks manually for the replaced physical disks with below commands. SCORE CellCLI> create celldisk CD_00_celadm01 lun=0_0 CellCLI> create celldisk CD_01_celadm01 lun=0_1 4 • Check that the Cell Disks are created and the status is normal CellCLI> list celldisk CD_00_celadm01 normal CD_01_celadm01 normal (………….. Output omitted for brevity) Now it’s time to create Grid Disks on the newly created Cell Disks. • Find out the Grid Disk name, size and offset by running the "list griddisk" command on System Disks from one of the surviving Cells. Example shown below: CellCLI> list griddisk attributes name,size,offset where celldisk=CD_00_celadm02 DATAC1_CD_00_celadm02 2.8837890625T 32M RECOC1_CD_00_celadm02 738.4375G 2.8838348388671875T CellCLI> list griddisk attributes name,size,offset where celldisk=CD_01_celadm02 DATAC1_CD_01_celadm02 2.8837890625T 32M RECOC1_CD_01_celadm02 738.4375G 2.8838348388671875T • Create Grid Disks based on the size from above example. Ensure that the Grid Disk with lower offset is created first for optimal performance. CellCLI> CREATE GRIDDISK DATAC1_CD_00_celadm01 CELLDISK = CD_00_celadm01, - SIZE = 2.8837890625T CellCLI> CREATE GRIDDISK RECOC1_CD_00_celadm01 CELLDISK = CD_00_celadm01, - SIZE = 738.4375G • Verify that Grid Disks are created correctly, using “list griddisk” command. CellCLI> list griddisk attributes name,size,offset,status where celldisk = CD_00_celadm01 DATAC1_CD_00_celadm01 2.8837890625T 32M active RECOC1_CD_00_celadm01 738.4375G 2.8838348388671875T active c. Now the user is required to log in to Oracle ASM instance and set the Disks to ONLINE using the following command for each disk group: SQL> ALTER DISKGROUP disk_group_name ONLINE DISKS IN FAILGROUP cell_name; If any system disks were replaced during the rescue procedure and Cell Disks and Grid Disks were recreated on them as explained in previous point, then the newly created Grid Disks need to be added to ASM using commands below: SQL>ALTER DISKGROUP DATAC1 ADD DISK 'o/*/DATAC1_CD_00_celadm01'; SQL>ALTER DISKGROUP RECOC1 ADD DISK 'o/*/RECOC1_CD_00_celadm01'; d. If flashcache and flashlog do not exist, create them by following these steps: cellcli> list flashlog cellcli> list flashcache Flashlog and flashcache can be created with the below commands: cellcli> create flashlog all cellcli> create flashcache all e. The cell can be reconfigured using the ALTER CELL command. We have presented an example for the most-common parameters below: CellCLI> ALTER CELL smtpServer='my_mail.example.com', - smtpFromAddr='[email protected]', - smtpFromPwd=email_address_password, - smtpToAddr='[email protected]', - notificationPolicy='critical,warning,clear', - notificationMethod='mail,snmp' f. The I/O Resource Management (IORM) plan and metric thresholds needs to be re-created. 5 Recovering a Cell Node using Internal USB In scenarios, where both USB and HDD are damaged, and there is no cell boot image created on external USB drive, a new external USB stick is used to recover the cell. Any USB stick of 4 GB or more can be used. The typical procedure steps are mentioned below: 1. The Vxxxxx.zip file from EDELIVERY needs to be downloaded and transferred to one of the compute nodes on the Exadata RACK and a new USB stick needs be inserted in the same compute node.