Dell SC Series Storage with Oracle ASM
Dell Engineering January 2016
A Dell Best Practices
Revisions
Date Description
May 2011 Initial release
April 2012 Content and format change
January 2016 Content and format change
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.
© 2016 Dell Inc. All rights reserved. Dell, the DELL logo, and the DELL badge are trademarks of Dell Inc. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims any proprietary interest in the marks and names of others.
2 Dell SC Series Storage with Oracle ASM | CML1111
Table of contents 1 Introduction ...... 5 1.1 Objective...... 5 1.2 Audience ...... 6 2 Technology Overview ...... 7 2.1 SC Series storage ...... 7 2.1.1 Features ...... 7 2.1.2 Benefits of Oracle databases on SC Series storage ...... 9 2.2 Oracle Technology ...... 9 2.2.1 Oracle ASM features ...... 9 2.2.2 Oracle Flex ASM features ...... 10 2.2.3 Oracle ASMLib Features ...... 12 2.2.4 Benefits of ASM, Flex ASM and ASMLib ...... 12 3 Components of ASM ...... 14 3.1 ASM Instance ...... 14 3.2 ASMLIb ...... 14 3.3 Disk Groups ...... 15 3.3.1 ASM Disk Group Redundancy ...... 15 3.3.2 ASM Recovery ...... 15 4 ASM best practices with Dell SC Series storage ...... 16 4.1 SAN and zoning ...... 16 4.2 SC LUN sizes and quantity...... 16 4.3 SC Series volume (LUN) creation for ASM disk groups ...... 16 4.4 SC Series storage profiles ...... 17 4.5 SC Series Replay profiles ...... 18 4.6 Multipath I/O ...... 21 4.7 ASM disk groups ...... 22 4.8 ASM disk group redundancy ...... 22 5 Conclusion ...... 23 A Flex ASM client query ...... 24 B Other useful commands for an ASM environment ...... 26 C Example: Oracle 12c with Flex ASM setup in UEK6 ...... 27
3 Dell SC Series Storage with Oracle ASM | CML1111
D 512 byte vs 4KB byte sectors ...... 42 E Additional resources ...... 43
4 Dell SC Series Storage with Oracle ASM | CML1111
1 Introduction Managing and provisioning storage for optimal configurations that accommodate the changing and increasingly demanding requirement of a growing Oracle® database is a key challenge for IT and vital to business success. Protecting a database and ensuring its integrity and availability is also mandatory for the enterprise and business.
When designing the physical structure of a database, database administrators must consider different storage configuration options. Storage solutions must facilitate high performance I/O and provide protection against failure of storage hardware such as disks and host bus adapters (HBAs). Growing and changing workloads require a dynamic storage configuration that provides a way to automate storage related tasks so that the risk of human error is reduced.
The Oracle Automatic Storage Management (ASM) feature simplifies storage provisioning and data layout challenges. The Dell™ Storage Center Thin Provisioning and Dynamic Block Architecture software can work in combination with ASM to allow Oracle data to be easily accessible, provide a Data Life Cycle management structure and have Dynamic Capacity in less time than other traditional storage solutions.
Harnessing the power of Oracle ASM with the intelligence of Dell SC Series storage yields a reliable, stable and dynamic infrastructure that can be protected with very little effort. Moreover, business applications can take advantage of all the disks in an SC Series storage solution.
1.1 Objective Although this document is not a comprehensive user guide, it provides some steps and general best practices to combine Oracle Automatic Storage Manager (ASM), Oracle Flex ASM, and Oracle unbreakable enterprise Linux® kernel (UEK6) with SC Series storage. After reviewing this document, the reader will be more informed on possible storage solutions and best practices to address Oracle database storage needs.
Keep in mind that the suggestions and possible solutions presented in this document should be reviewed and adjusted for individual environments. Dell advises customers to understand the type of storage needs required by the business before designing and implementing a solution.
For detailed information on any part of the Oracle installation and configuration used in this paper, refer to the information provided at the My Oracle Support site. Oracle ASM tuning, converting between standard legacy ASM deployments and Flex ASM deployments, Flex clusters, and Oracle proxy ASM Instance are beyond the scope of this document. See the reference in Appendix E at the end of this document for additional information.
5 Dell SC Series Storage with Oracle ASM | CML1111
1.2 Audience This document is intended for DBAs, system administrators, storage administrators, and architects seeking to design, implement and maintain robust database and storage systems for Oracle database solutions that are effective and efficient. Readers should have prior experience in operating the following:
RAID, Fibre Channel, Serial Attached SCSI (SAS), LUNs, multipathing SC Series Storage Dell Enterprise Manager General IP networking concepts Familiarity with Oracle Linux and administration Oracle 11g, and 12c databases and architecture Oracle Grid Infrastructure Oracle real application clusters (RAC) or single instance database Oracle ASM and Flex ASM
6 Dell SC Series Storage with Oracle ASM | CML1111
2 Technology Overview The storage component of an infrastructure can be managed with very little effort by combining Dell SC Series storage with Oracle ASM.
Evaluate both components and assess the benefits before deciding on the options that will be the best fit for the platform and business expectations.
2.1 SC Series storage SC Series storage provides a number of features that can be used in an Oracle database environment to create a robust storage solution and assist DBAs with data tiering, database performance, efficient use of storage and database activities.
2.1.1 Features Dynamic block architecture: Records and tracks metadata for every block of data and provides system intelligence on block usage. The metadata enables Storage Center to take a more sophisticated and intelligent approach to storing, recovering and managing data.
Storage virtualization: Virtualize enterprise storage at the disk level, creating a dynamic pool of storage resources shared by all servers. Because read/write operations are spread across all available drives within the same tier, multiple requests are processed in parallel, boosting system performance.
Dynamic Capacity (Thin Provisioning): This feature delivers the highest storage utilization possible by eliminating allocated but unused capacity. Dynamic Capacity completely separates storage allocation from utilization, enabling users to create any size virtual volume upfront, and only consumes actual physical capacity when data is written by the application
Data Instant Replay (DIR): DIR is a snapshot technology that provides continuous, space-efficient data protection using Replays. Replays create point-in-time copies of volumes; further changes to the volumes are recorded in a way that allows the volume to be rolled back to its original state. There are various uses for Replays in an Oracle environment. Oracle database backups or recoveries, and database cloning are easily performed when using consistent replay profiles.
7 Dell SC Series Storage with Oracle ASM | CML1111
Figure 1 SC Series consistent replay
Data Progression (DP): This is a fluid data storage or automated tiered storage feature. Data Progression automatically migrates enterprise data to the optimal storage tier based on a set of predefined or custom policies, or storage profiles. Data Progression eliminates the need to manually classify and migrate data to different storage tiers while reducing the number and cost of drives and reducing the cooling and power costs.
Fast Track: This technology enhances Automated Tiered Storage by dynamically placing the most frequently accessed data on the fastest, or outer, tracks of each disk drive. Meanwhile, the least active blocks of data remain on the inner tracks of all the drives in each storage tier.
For additional documentation on these features, see references in Appendix E.
8 Dell SC Series Storage with Oracle ASM | CML1111
2.1.2 Benefits of Oracle databases on SC Series storage Table 1 Benefits of Oracle on SC Series storage Benefit Details
Lower total cost of ownership (TCO) Reduces acquisition, administration, and maintenance costs
Greater manageability Ease of use, implementation, provisioning, and management
Simplified RAC Implementation Provides shared storage (raw or file systems)
High availability and scalability Clustering provides higher levels of data availability and combined processing power of multiple server for greater throughput and scalability
Information Life Cycle (ILM) benefits Provides tiered storage, dynamic capacity, data progression, thin provisioning, instant replay (snapshot) and more
Expand database buffer cache beyond Provides the ability to extend Database Smart Flash Cache the SGA in main memory using Dell SC TIER 0 storage (Server-side cache)
2.2 Oracle Technology ASM is an alternative to other conventional volume managers, file systems and raw devices. It provides a simplified storage management interface for DBAs. Roughly speaking, ASM is a logical volume manager that virtualizes storage to a set of disk groups and provides redundancy options to enable a high level of protection. Oracle recommends ASM solution for storage management.
Flex ASM is an enhanced version of ASM that is offered in Oracle 12c. Flex ASM provides the capability to change the architecture of the ASM cluster, and provides a way to mitigate database instance outages should the ASM instance running on the database server sustain an outage.
With respect to the physical storage, there is no difference on how storage should be configured for ASM or Flex ASM. The configuration can be the same. The difference between ASM and Flex ASM is how ASM is configured. With no difference in physical configuration, between ASM or Flex ASM implementation, the use of Dell SC Series storage features remains the same between the ASM implementations.
2.2.1 Oracle ASM features Oracle ASM was introduced in Oracle 10g and provides a vertical integration of the file system and a volume manager specifically built for only Oracle database files. ASM can assemble multiple disk devices that behave as block devices into logical containers called disk groups, and can perform non-intrusive storage configuration, striping and mirroring within the disk groups with automatic rebalancing. ASM distributes I/O loads across all available disk devices within disk groups to optimize performance and resource utilization, while removing the need for manual I/O tuning by spreading out the database files to
9 Dell SC Series Storage with Oracle ASM | CML1111
avoid hotspots. ASM also assists DBAs manage a dynamic database environment by allowing database growth and transparent data relocation among the disk devices without having to shut down the database to adjust storage allocation.
ASM nodes (DB and ASM instance)
Node 1 Node 2 Node 3 Node 4 DB1 DB1 DB1 DB2 DB2
ASM ASM ASM ASM
Oracle Fabric Interconnect
DATA FRA
Figure 2 Traditional ASM configuration with dedicated ASM instance for each database server
2.2.2 Oracle Flex ASM features Prior to the availability of Flex ASM, an ASM instance is required on each database server where there is a database instance. This creates the possibility of a single-point of failure with the ASM architecture on any one of the database servers. For example, if an ASM instance fails, so would the corresponding database instance.
Flex ASM mitigates the issue of a database instance outage caused by an outage of the corresponding ASM instance on the same database server. The mitigation is accomplished by Oracle starting another ASM instance in the cluster so that the ASM cardinality is fulfilled and reroutes ASM traffic to an available ASM instance in the cluster. The selected ASM instance is the one with the least amount of activity. Rerouting ASM traffic resolves the single-point of failure in the event that an ASM instance fails and provides highly scalable, massively parallel and highly available framework. ASM traffic can reside on either the private Oracle RAC interconnect, or an optional dedicated network for ASM traffic. The number of nodes running an ASM instance in a Flex ASM configuration is three by defaut, but is controlled by the ASM cardinality and can be altered.
10 Dell SC Series Storage with Oracle ASM | CML1111
ASM Flex nodes ASM client (DB and ASM instance) (DB but no ASM instance)
Node 1 Node 2 Node 3 Node 4 DB1 DB1 DB1 DB2 DB2
ASM ASM ASM
Fabric
Oracle Interconnect Optional ASM Network DATA FRA
Figure 3 Flex ASM architecture with four database servers with ASM cardinality of three
ASM Flex nodes ASM client (DB and ASM instance) (DB but no ASM instance)
Node 1 Node 2 Node 3 Node 4 DB1 DB1 DB1 DB2 DB2
ASM ASM ASM ASM Fabric
Oracle Interconnect Optional ASM Network DATA FRA
Figure 4 Flex ASM architecture with rerouted ASM traffic
All SC Series LUNs intended to be used in the environment must be presented on all ASM Flex nodes and ASM client nodes intended to be used in the environment. This applies to a node not currently running a database instance, nor an ASM instance, but may at some point be used by flex ASM as an ASM instance failover node.
11 Dell SC Series Storage with Oracle ASM | CML1111
2.2.3 Oracle ASMLib Features Introduced in 2004, ASMLib is an optional set of tools and kernel driver that can be inserted between ASM and the hardware, as well as an application library used by the Oracle database software to access ASM disks. It is a support library for the ASM feature of Oracle database servers as well as RAC installations. ASM and regular database instances can use ASMLib as an alternative interface for disk access yielding greater performance and stability compared to equivalent workloads on standard Linux file systems. Oracle recommends implementing ASM with ASMLib together for better manageability and persistent device naming, but does not make claims that ASM with ASMLib delivers performance benefits over ASM without ASMLib.
ASMLib has three components:
Kernel driver (oracleasm): Oracle ASMLib Linux kernel driver that is open-sourced (GPL) and is available from Oracle as source and binary RPMs. Although is it open-source, it has not been accepted into the mainline Linux kernel. Support tools (oracleasm-support): an open-source package (GPL) containing utilities to manage the ASMLib Linux kernel driver. Available from Oracle as source and binary RPMs. Application library (oracleasmlib package): ASM library. This is a closed source, binary-only RPM, available as a free download from Oracle.
ASMLib addresses deficiencies that existed in 2.4 Linux kernel such as the lack of async and direct I/O. These deficiencies were addressed by the Linux community in the 2.6 Linux kernel with the addition of udev and device-mapper multipath which together with LVM provided native multipathing, scalable native volume management, and persistent device naming.
Although it is recommended to use ASMLib, it is not necessary. Should it not be used, udev rules and/or using device mapper multipathing must be used to achieve persistent device naming.
2.2.4 Benefits of ASM, Flex ASM and ASMLib Some benefits of using ASM, Flex ASM and ASMLib are listed below. For additional benefits, please refer to the references at the end of this document.
Table 2 Some benefits of Oracle ASM and ASMLib Benefit Details
Provides easier administration and lower Provides ease of use, storage management, and total cost of ownership (TCO) management and discover of correct devices for ASM allowing administrators more time to concentrate on other important tasks and reduce the overlap of responsibilities
Device name persistence Eliminates the management of custom udev rules for ASM disk devices
Greater manageability Ease of use, implementation, provisioning, and management
12 Dell SC Series Storage with Oracle ASM | CML1111
Benefit Details
Simplified RAC Implementation Provides shared storage (raw or file systems)
Recommended by Oracle ASM is Oracle preferred logical volume manager providing greater efficiencies and optimizations when working with database files via direct and async I/O provided by the ASMLib kernel driver
Greater data reliability ASM provides data mirroring on a file bases rather than on a volume basis. Hence, the same disk group can contain a combination of files protected by mirroring, parity, or not protected at all.
Improved performance Automatic distribution of database files across all disks in a disk group. Delivers the performance of raw disk I/O without the inconvenience of managing raw disks
Reduced overhead Eliminates the overhead at the file system layer as I/O is passed from the database directly to the disks
Increased database uptimes Unlike logical volume managers (LVMs), ASM maintenance operations do not require a database outage
Support for multiple stripe sizes ASM provides the ability to use different strip sizes for different database file types
Simplified RAC Implementation Provides shared storage (raw or file systems)
Reduced time to resync data after a failure ASM fast mirror resync reduces the amount of time required to resynchronize a disk after a failure by journaling the ASM extents that are modified during the outage and applying them after the failure is resolved
13 Dell SC Series Storage with Oracle ASM | CML1111
3 Components of ASM Disk groups are the primary component of ASM. Once they are created, they can be specified as the default location for database files created in the database. Oracle has several different interfaces available for the creation and management of disk groups, their contents, and their metadata.
A disk group consists of a grouping of disks that are managed together as a unit. These disks are referred to as ASM disks. Files written on ASM disks are ASM files, whose names can be automatically generated by ASM. However, user-friendly alias names for ASM files can be used, but a hierarchical directory structure for these alias names must be created.
3.1 ASM Instance Starting with Oracle 10g and above, there are two types of instances: database and ASM. The ASM instance, which is generally named +ASM, is started with the INSTANCE_TYPE=
3.2 ASMLIb As mentioned above, Oracle developed an optional storage management API interface for block device management, discovery, and provisioning called ASMLib that provides storage vendors the ability to supply a plug-in library to extend storage-related features and exploit storage array capabilities when using ASM. It is simply an add-on module that simplifies the management and discovery of ASM disks and provides an alternative to the standard operating system interface for ASM to identify and access block devices.
ASMLib is not required to run ASM, but if ASMLib is not used, native OS methods (like UDEV rules) must be used to manage the ASM block devices. UDEV rules can be defined in /etc/udev/rules.d/99- oracle-asmdevices.rules. For example:
KERNEL=="sdb", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", OWNER="grid", GROUP="oinstall", MODE="0660"
ASMLib also allows a database to use more efficient access to the logical volume groups it is using.
If ASMLib is used, the user and group permissions will be set correctly on the devices under /dev/oracleasm/disks/*. The values of user and group are configured while running /etc/init.d/oracleasm configure.
14 Dell SC Series Storage with Oracle ASM | CML1111
3.3 Disk Groups Once the ASM disk devices are discovered, a disk group can be created that will encapsulate one or more of these disks. A disk group, which is the highest-level data structure in ASM, is comparable to an LVM volume group. However, there are several differentiators between typical LVM volume groups and ASM disk groups.
An ASM file system layer is implicitly created within a disk group. This file system is transparent to users and only accessible through ASM, interfacing databases, and ASM command line tool. There are inherent automatic file-level striping and mirroring capabilities. A database file created within an ASM disk group will have its file extents (not database extents) distributed equally across all online disks in the disk group, which provides an even I/O load.
Oracle recommends that generally no more than two disk groups be maintained and managed per single ASM instance or RAC cluster. Data that can be categorized with different storage characteristics can be stored in distinct disk groups with different redundancy settings and fail-groups if necessary. It is recommended to have at least the following to disk groups:
Database data (DATA): Where active database files such as data files, control files, online redo logs, archive redo logs, and change tracking files used in incremental backups are stored. Flash Recovery Area (FRA): Where recovery-related files are created, such as multiplexed copies of the current control file and online redo logs, backup sets, and flashback log files.
3.3.1 ASM Disk Group Redundancy There are three types of redundancy for ASM disk groups:
Normal Redundancy: Provides 2-way mirrored which requires two ASM failure groups. This is the default. High Redundancy: Provides 3-way mirrored which requires three ASM failure groups. External Redundancy: Provides no protection within ASM. This level of redundancy requires RAID protection on the storage arrays. This is generally the recommended redundancy level when using SC Series storage. There are use cases in Oracle ASM that might require either normal or high redundancy even if SC Series storage is used. One such use case is Oracle extended distance clusters.
3.3.2 ASM Recovery Since ASM manages the physical access to ASM files and its metadata, a shutdown of the ASM instance will cause the client database instances to shutdown as well, unless Flex ASM is used. In a Flex ASM implementation, a shutdown of the ASM instance will not cause the database instance to shutdown.
In a single ASM instance configuration, if the ASM instance fails while disk groups are open for update, after the ASM instance restarts, it recovers transient changes when it mounts the disk groups.
15 Dell SC Series Storage with Oracle ASM | CML1111
4 ASM best practices with Dell SC Series storage This section provides some general best practices when using ASM with Dell SC Series storage. Incorporating them into a solution will be determined by specific business and infrastructure requirements.
4.1 SAN and zoning Dell recommends configuring the arrays with two controllers and multiple front-end ports zoned across two separate VSANS. This two controller configuration with multiple front-end ports is recommended to avoid a single point of failure should a controller fail. Virtual port mode is also a recommended configuration in storage.
For additional information on virtual port mode, see the Dell Storage Center System Owner’s Manual.
4.2 SC LUN sizes and quantity LUNs sizes can vary depending on the size of the database. Generally, Dell does not recommend any LUN size greater than 1TB. If a database is roughly 1TB in size, 4 x 250GB LUNs might be a good number of LUNs to start with for the ASM disk groups.
4.3 SC Series volume (LUN) creation for ASM disk groups With traditional storage arrays that do not have virtualization, thin provisioning and Dynamic Capacity, multiple LUNs need to be created in the array and mapped to a database server as block devices in order to use the SAME (Stripe and Mirror Everything) methodology provided by ASM. While ASM recommends multiple disks of the same capacity per ASM disk group for I/O performance, the multiple disk recommendation is no longer necessary on SC Series arrays because the virualization and SAME methodology of disk management is provided in the storage.
When not using ASMLib, Dell recommends using an entire disk as an ASM disk rather than partitioning the disk and using a the non-primary partition as an ASM disk. Attempting to expand the size of a disk that is not the primary partition requires advanced knowledge of Linux systems and Oracle backups and should only be done after careful planning and consideration. This includes making sure a valid backup exist of the disks prior to expanding their size. When looking at the recommendation of the disk backup as it applies to an Oracle environment, the backup may need to be applied against the entire Oracle database. For example, if an Oracle database consists of 20 disks and only one disks needs to be increased in size, the backup needs to include a backup of all 20 disks.
Therefore, when not using ASMLib, use an entire disk and not a partion as an ASM disk to ease the administration and management of the disk should the disk increase in size. Additional information on resizing a LUN can be found in the Dell Storage Center with Red Hat Enterprise Linux (RHEL) 6x Best Practices document on TechCenter. If ASMLib is used, then the disk must be partitioned per Oracle requirements. The first partition will be used by ASMLib to save its metadata, and the remaining portion of the disk will be used by the database.
16 Dell SC Series Storage with Oracle ASM | CML1111
With SC Series storage features such as virtualization, thin provisioning and Dynamic Capacity, an Oracle database only needs to have 4 LUNs created in a SC array. Two of the LUNs will be presented to the database server as block devices and initialized as ASM disks and assigned to the ASM disk group for database data. The remaining two LUNs will be presented to the database server as block devices and initialized as ASM disks and assigned to the ASM disk group for the Flash Recovery Area. Having fewer disks per ASM disk group minimizes the administration headaches for DBAs, system administrators and storage administrators. If necessary, additional LUNs could be created and used as ASM disks and assigned to the same disk groups, or even added to additional disk groups. Having the ability to have multiple disks provides a way to take advantage of multiple disk queues at the operating system level and helps if the HBA is not saturated. If multiple LUNs are created with the intent of usging them for ASM disk groups and the Instant Replay feature, make sure to configure a Consistent Replay Profile and apply it to all the ASM LUNs for a given database.
The four LUN minimum is not only an Oracle recommendation, but Dell recommends an even number of LUNs in a dual controller SC Series array. An even number of LUNs will be automatically balanced across both controllers so that there is an even distribution of I/O activity across the controllers.
4.4 SC Series storage profiles Storage profiles define the tiers of storage including the type of redundancy used. When initially configuring the arrays, a set of default storage profiles are available. These default storage profiles cannot be modified, but customized storage profiles can be created should the need arise. For a complete list of default storage profiles and their configurations, see document see the Dell Storage Center System Manager 6.6 Administrator’s Guide provided with the SC Series storage purchase.
For most Oracle implementations, Dell recommends using one of the standard storage profiles. In an all flash optimized array, Dell recommends using either the Flash Optimzied with Progression, or Flash Only with Progression storage profile. If a non-flash optimized array is used, Dell recommends using the Recommended (All Tiers) storage profile. Any of these profiles might be good for initial use, however a Dell storage architect may recommend using a different standard profile, or a custom profile for handling specific types of data and processing.
Figure 5 Available default storage profiles in Storage Center 6.5 when no SSDs are used
17 Dell SC Series Storage with Oracle ASM | CML1111
Figure 6 System Manager showing storage profile assignment to a volume
4.5 SC Series Replay profiles If a LUN happens to contain all the physical files (such as control file, redo logs, data, index, system, temporary and undo) of an Oracle database, it is sufficient to create a snapshot of that single LUN and have a crash recovery image of the database. Although it is sufficient to generate a snapshot of the database in this particular case, it is extremely unlikely that this will always be the case for the life of the database. For example, what happens if another LUN is needed because the initial LUN has reached 100 percent capacity? Because of the uncertainty of the future number of LUNs required by the database and the need to have a crash consistent image of all the underlying volumes, it’s recommended to never create a snapshot of a database without a user-defined consistent Replay profile
In addition to a user defined consistent Replay profile, the database must also be placed in BEGIN BACKUP mode prior to the creation of the snapshot. Once the snapshot is created, the database can be taken out of BEGIN BACKUP mode.
Within the SC Series Storage Manager or Dell Enterprise Manager, the Replay profile types can easily be identified by the icon associated with the Replay profile.
18 Dell SC Series Storage with Oracle ASM | CML1111
Default Replay profiles
Custom Replay profiles
Custom consistent Replay profiles
Figure 7 Replay Profile types and icons in SC System Manager
Default Replay profiles
User-defined Replay profiles
User-defined consistent Replay profiles
Figure 8 Replay Profile types and icons in Dell Enterprise Manager
It is recommended to always use a consistent Replay profile for Oracle, regardless of the number of LUNs used by the database. Any snapshot generated from the consistent Replay profile while the database is in BEGIN BACKUP mode could then be used for rapid database cloning, database backup and recovery operations, and database replication.
19 Dell SC Series Storage with Oracle ASM | CML1111
Consistent Replay profile Two LUNs used by ASM
Figure 9 A Consistent Replay profile consisting of two LUNs used by ASM
To create a consistent Replay profile, right-click the Replay Profiles folder and select Create Consistent Replay Profile from the context menu and follow the steps presented by the System Manager wizard
Figure 10 Creating a Consistent Replay Profile
For additional information on creating Consistent Replay profiles, see the Dell Storage Center System Manager 6.6 Administrator’s Guide provided with the SC series storage purchase.
20 Dell SC Series Storage with Oracle ASM | CML1111
4.6 Multipath I/O An I/O path generally consists of an initiator port, fabric port, target port, and LUN. Each permutation of this I/O path is considered an independent path. Dynamic multi-pathing/failover tools aggregate these independent paths into a single logical path. This path abstraction provides I/O load balancing across the host bus adapters (HBA), as well as non-disruptive failovers on I/O path failures. The use of dynamic multi- path I/O combined with ASM and SC Series storage technologies such as virtualization and thin provisioning yields the best high availability and performance for a database server.
Fount-end redundancy is recommended to eliminate single points of failure between a server and storage. This redundancy was accomplished with configuring and using multipath. The code below is a sample of a multipath.conf file used in a configuration.
defaults { udev_dir /dev polling_interval 5 path_grouping_policy failover getuid_callout "/lib/udev/scsi_id --whitelisted -- device=/dev/%n" prio const path_checker directio rr_min_io 1000 rr_weight uniform failback manual no_path_retry fail user_friendly_names yes } devices { device { vendor "COMPELNT" product "Compellent Vol" features "0" hardware_handler "0" no_path_retry fail } } blacklist { devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^hd[a-z]*" devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]" }
For additional multipath information, see the My Oracle Support (MOS) document 1538626.1, and Dell Storage Center System Owner’s Manual, Dell Storage Center Deployment Guide, and DELL Storage Center with Red Hat Enterprise Linux (RHEL) 6x Best Practices at (http://en.community.dell.com/techcenter/extras/m/white_papers/20437964/download)
21 Dell SC Series Storage with Oracle ASM | CML1111
4.7 ASM disk groups Oracle recommends that generally no more than two disk groups are maintained and managed per single ASM instance or RAC cluster. Data that can be categorized with different storage characteristics can be stored in distinct disk groups with different redundancy settings and fail-groups if necessary. Examples of disk groups are:
Database Area Flash Recovery Area
4.8 ASM disk group redundancy ASM external redundancy disk groups were designed to leverage RAID protection provided by storage arrays. Dell SC Series storage supports several types of protection against loss of media and provides transparent failover in the event of a specific disk or component failure. Since ASM does not provide redundancy when external redundancy is used, the server CPU and other resources are less consumed, which means there is an increase of available CPU cycles for database operations.
SC Series storage provides redundancy at the storage (disks) level. If a database server cannot access the ASM LUN at the operating system level due to operating system errors, then the ASM instance will fail. If this is a primary concern, then configure the ASM disk group with normal redundancy (i.e. with 2 failure groups), or high redundancy. If extended distance clusters are a primary objective, then either normal or high redundancy levels must be chosen, with either ASM or Flex ASM. Otherwise, Dell recommends choosing external redundancy on the ASM disk group.
22 Dell SC Series Storage with Oracle ASM | CML1111
5 Conclusion In summary, ASM provides a simplified storage management interface for DBAs while Dell SC Series storage technologies provide ease of use for storage provisioning and management for the system administrators. These technologies simplify the management of database resources and storage resources. This allows DBAs and system administrators more time to concentrate on other important tasks and reduce the overlap of responsibilities.
23 Dell SC Series Storage with Oracle ASM | CML1111
A Flex ASM client query
The following 12c query shows what database instances are being serviced by the ASM instance that executed the below query:
set linesize 180 set pagesize 60
col group_number format 99 heading "Grp|Num" col instance_name format a8 heading "Inst|Nm" col db_name format a8 heading "DB|Name" col cluster_name format a12 heading "Clu|Nm" col status format a9 heading "Status" col state format a7 heading "State" col name format a7 heading "DG Name" col type format a6 heading "ASM|Redun-|dancy" col total_mb format 999,999 heading "Total|MB" col allocation_unit_size format 9,999,999 heading "Alloc|Unit|Sz" col offline_disks format 99 heading "Offline|Disks" col voting_files format a7 heading "Voting|Files"
select ac.group_number , ac.instance_name , ac.db_name , ac.cluster_name , ac.status , ac.con_id , adg.sector_size , adg.block_size , adg.allocation_unit_size , adg.state , adg.type , adg.total_mb , adg.offline_disks , adg.voting_files from v$asm_diskgroup adg , v$asm_client ac where adg.group_number = ac.group_number order by 1, 2;
Sample output of the above query:
Alloc ASM Grp Inst DB Clu Unit Redun- Total Offline Voting Num Nm Name Nm Status Sz State dancy MB Disks Files
24 Dell SC Series Storage with Oracle ASM | CML1111
------1 +ASM1 +ASM rac-cluster CONNECTED 1,048,576 MOUNTED EXTERN 15,358 0 Y 1 -MGMTDB _mgmtdb rac-cluster CONNECTED 1,048,576 MOUNTED EXTERN 15,358 0 Y 2 +ASM1 +ASM rac-cluster CONNECTED 1,048,576 MOUNTED EXTERN 511,993 0 N 2 prd1 prd rac-cluster CONNECTED 1,048,576 MOUNTED EXTERN 511,993 0 N 2 prd2 prd rac-cluster CONNECTED 1,048,576 MOUNTED EXTERN 511,993 0 N 3 prd1 prd rac-cluster CONNECTED 1,048,576 MOUNTED EXTERN 511,993 0 N 3 prd2 prd rac-cluster CONNECTED 1,048,576 MOUNTED EXTERN 511,993 0 N
7 rows selected.
The output shows that instance +ASM1 is servicing three database instances:
1. -MGMTDB 2. prd1 3. prd2
But a quick check for pmon processes on the server shows that database instances +ASM1 and prd1 are the only instances running on the server.
ps -ef | grep pmon | grep –v grep grid 102713 1 0 10:36 ? 00:00:00 asm_pmon_+ASM1 oracle 103746 1 0 10:37 ? 00:00:00 ora_pmon_prd1
Instances prd2 and –MGMTDB are running on a different server.
25 Dell SC Series Storage with Oracle ASM | CML1111
B Other useful commands for an ASM environment
The following commands can be useful in an ASM configuration. For a complete list of commands, refer to the references in Appendix E: Additional resources
Table 3 Helpful commands for ASM configuration Command Description
srvctl modify asm -count
srvctl config asm Display the ASM configuration
srvctl status asm -detail Display status of ASM on all nodes
crsctl status res ora.asm [-t|–l|- Display status of ASM and cardinality of Flex ASM if f] enabled.
crsctl status res Display status of the listener used by Flex ASM ora.ASMNET1LSNR_ASM.lsnr [-t|-l|-f]
asmcmd showversion Display ASM version
asmcmd showpatches Display the patches applied on the grid home
asmcmd showclustermode Show if flex ASM is enabled
asmcmd showclusterstate Displays ASM rolling migration/patching state
asmcmd lsct
asmcmd pwget --asm Display ASM password file location
asmcmd spget Display ASM spfile file location
asmcmd shutdown Shut down an Oracle ASM instance
asmcmd startup Start up an Oracle ASM instance
asmcmd spbackup Back up an Oracle ASM spfile
oifcfg getif Display the network interfaces (including ASM’s)
olsnodes –n –s -a Display state and type of nodes (hub,leaf) in cluster
See Oracle Automatic Storage Management, Administrator’s Guide, 12c Release 1 (12.1) for additional information.
26 Dell SC Series Storage with Oracle ASM | CML1111
C Example: Oracle 12c with Flex ASM setup in UEK6
Below is an example of installing and configuring Flex ASM 12c on UEK6.5 that’s connected to a Dell SC array in dual controller mode using virtual port mode that is providing storage to a Linux system using native multi-path. For detail Oracle installation and configuration instructions, refer to Oracle documentation.
1. When installing UEK 6.5 in the lab environment, the linux package group Desktop was selected as recommended by My Oracle Support (MOS) note 401167.1. The following additional packages were installed but were not necessary:
Table 4 Additional Linux packages Package Group Packages
Base System Compatibility libraries
Legacy UNIX compatibility
Storage Availability Tools
iSCSI Storage Client
Development Additional Development
Desktop Platform Development
Development tools
Eclipse
Server Platform Development
2. Linux graphical boot was disable and grub install timeout was increased:
cp -p /etc/grub.conf /etc/grub.conf.`date +%Y%m%d_%H%M` (echo "/^timeout/s/5/10/ /vmlinuz-3.8.13-16.2.1.el6uek.x86_64/s/ rhgb quiet// /vmlinuz-2.6.32-431.el6.x86_64/s/ rhgb quiet// wq ") | ex -s /etc/grub.conf shutdown -r now
3. After creating the necessary LUNs in SC and presenting them to the servers, the new devices were located on the servers with the following:
for a in $(ls -1 /sys/class/scsi_host) do echo "- - -" > /sys/class/scsi_host/$a/scan done
27 Dell SC Series Storage with Oracle ASM | CML1111
4. The necessary device-mapper packages should be installed by default, but the multipath modules had to be loaded into the kernel.
modprobe dm-multipath modprobe dm-round-robin
5. The multipath daemon was added to the system as a new service, and all binding and wwid files were removed
chkconfig multipathd on chkconfig --add multipathd chkconfig --list multipathd /etc/init.d/multipathd status /etc/init.d/multipathd stop rm -f /etc/multipath/bindings rm -f /etc/multipath/wwids
6. multipath.conf was updated on all servers with the new devices and multipath started
vi /etc/multipath.conf /etc/init.d/multipathd start
7. Disk devices were formatted and others partitioned for ASMLib. Partition changes were loaded into the kernel.
mkfs -t ext4 /dev/mapper/orabin fdisk /dev/mapper/crs1 n p 1
fdisk /dev/mapper/asm-data1 n p 1
fdisk /dev/mapper/asm-data2 n p 1
28 Dell SC Series Storage with Oracle ASM | CML1111
w fdisk /dev/mapper/asm-fra1 n p 1
fdisk /dev/mapper/asm-fra2 n p 1
partprobe
8. Set the domainname in Linux
echo "domainname techsol.local" >> /etc/rc.local domainname techsol.local
9. /etc/hosts was updated with entries for all RAC nodes, Flex ASM client nodes, the RAC private interconnect, and VIPs. 10. Disable Linux NetworkManager
chkconfig --list NetworkManager service NetworkManager stop chkconfig NetworkManager off chkconfig --list NetworkManager
11. Change the network interface scripts so they are not controlled by NetworkManager
for i in `grep -i "^NM_CONTROLLED" /etc/sysconfig/network-scripts/ifcfg* | grep -i yes | cut -d: -f1` do (echo "/^NM_CONTROLLED/s/yes/no/ wq ") | ex -s $i done
12. Change the network interface boot protocol to none
for i in `grep -i "^BOOTPROTO=" /etc/sysconfig/network-scripts/ifcfg* | cut -d: -f1` do (echo "/^BOOTPROTO=/s/dhcp/none/
29 Dell SC Series Storage with Oracle ASM | CML1111
wq ") | ex -s $i done
13. Configure a network interface on all nodes for the private RAC interconnect and Flex ASM network
(echo "/^ONBOOT=/s/no/yes/ wq ") | ex -s /etc/sysconfig/network-scripts/ifcfg-
14. Restart the network service on all nodes
service network restart service network status
15. Create a mount point for Oracle binaries and add it to /etc/fstab
mkdir /u01 echo "/dev/mapper/orabin /u01 ext4 defaults 0 0" >> /etc/fstab mount -a df -k chmod 777 /u01
16. Install ASMLib components.
Since this is UEK 6, there is no need to install the kernel driver as it is built into the kernel. If however this were OEL 6, the ASMLib kernel driver would need to be installed because it is not prebuilt into the kernel.
1. Install oracleasm-support.
yum install oracleasm-support.x86_64
2. Get the Oracle Linux 6 ASMLib package from http://www.oracle.com/technetwork/server- storage/linux/asmlib/ol6-1709075.html and install it.
rpm -ihv oracleasmlib-2.0.12-1.el6.x86_64.rpm
3. Download the latest Dell Oracle deployment tar file from:
http://en.community.dell.com/techcenter/enterprise-solutions/w/oracle_solutions/4957.oracle- database-deployment-automation.aspx
The deployment tar contains a package that creates users grid and oracle, sets file handler limits for gird and oracle, set kernel memory parameters, and creates a /u01 directory structure for grid and oracle.
1. Save the tar file to each of the nodes in the cluster, untar it, and execute the components
tar -zxvf dell-oracle-deployment-EL6-12cR1-2014.02-3.tar.gz yum -y install dell-oracle-rdbms-server-12cR1-preinstall-1.0- 12.el6.x86_64.rpm --nogpgcheck yum -y install dell-oracle-utilities* --nogpgcheck
30 Dell SC Series Storage with Oracle ASM | CML1111
dodeploy -r 12cR1 -g -s
4. Install x11 apps and fonts
yum install xorg-x11-apps yum install xorg-x11-fonts*
5. Disable SELinux as ASMLib is not compatible with it
sestatus getenforce echo 0 > /selinux/enforce sestatus getenforce cat /etc/selinux/config (echo "/^SELINUX=disabled/s/disabled/permissive/ wq ") | ex -s /etc/selinux/config cat /etc/selinux/config
6. Disable firewalls for ASMLib
service iptables save service iptables stop chkconfig iptables --list chkconfig iptables off chkconfig iptables --list service iptables status
service ip6tables save service ip6tables stop chkconfig ip6tables --list chkconfig ip6tables off chkconfig ip6tables --list service ip6tables status
7. Set kernel panic for 12c. See MOS note 1529864.1 for more information
echo "kernel.panic_on_oops=1" >> /etc/sysctl.conf sysctl -p
8. Change Linux security limits for Oracle and Grid
echo " grid soft nofile 131072 grid hard nofile 131072 grid soft nproc 131072 grid hard nproc 131072 grid soft stack 10240 grid hard stack 32768
31 Dell SC Series Storage with Oracle ASM | CML1111
oracle soft nofile 131072 oracle hard nofile 131072 oracle soft nproc 131072 oracle hard nproc 131072 oracle soft core unlimited oracle hard core unlimited oracle soft memlock unlimited oracle hard memlock unlimited" >> /etc/security/limits.conf
9. Change Linux’s system profile
echo 'if [ $USER = "oracle" ] || [ $USER = "grid" ] ; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi fi' >> /etc/profile
10. Disabled OS network time protocol daemon
mv /etc/ntp.conf /etc/ntp.conf.orig
11. Set NOZEROCONF in the sysconfig network script and restart network services from the console.
echo "NOZEROCONF=yes" >> /etc/sysconfig/network cat /etc/sysconfig/network
12. Restart network services from the console.
service network restart
13. Configure ASMLib. Enter the user and group owner for the user owning the ASM binaries. Since the lab environment was using 12c, this user is the Grid user. This will configure the on-boot properties of the ASM library driver, will determine whether the driver is loaded on boot, what permissions it will have, and whether it should scan for ASM disks on boot.
/etc/init.d/oracleasm configure
14. 12c oracleasm configure will ask the following questions. Current values will be shown in brackets. Pressing ENTER without typing an answer will keep the current value. Ctrl-C will abort the configuration process.
Default user to own the driver interface []: grid Default group to own the driver interface []: oinstall Start Oracle ASM library driver on boot (y/n) [y]: Scan for Oracle ASM disks on boot (y/n) [y]: Writing Oracle ASM library driver configuration: done Initializing the Oracle ASMLib driver: [ OK ]
32 Dell SC Series Storage with Oracle ASM | CML1111
Scanning the system for Oracle ASMLib disks: [ OK ]
15. The remaining 12c ASMLib configuration parameters were set according Oracle best practices for ASMLib. See http://www.oracle.com/technetwork/topics/linux/multipath-097959.html for additional information.
(echo "/^ORACLEASM_SCANORDER/s/""/"dm-"/ wq ") | ex -s /etc/sysconfig/oracleasm
(echo "/^ORACLEASM_SCANEXCLUDE/s/""/"sd"/ wq ") | ex -s /etc/sysconfig/oracleasm
(echo "/^ORACLEASM_USE_LOGICAL_BLOCK_SIZE/s/"false"/"true"/ wq ") | ex -s /etc/sysconfig/oracleasm
ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size reported by SCOS instead of the physical block size. The default is 'false'. By default, SCOS 6.x reports the logical block size of 4KB.
See appendix D 512 byte vs 4KB byte sectors” for a discussion of 512 byte vs 4K byte sectors reported by Dell SC Series storage and the significance it has with Oracle.
16. Set the passwords to the grid and oracle users.
passwd oracle << EOF oracle oracle EOF
passwd grid << EOF oracle oracle EOF
17. Create staging directories for grid and Oracle software.
mkdir -p /u01/sw_grid chown grid:oinstall /u01/sw_grid chmod 777 /u01/sw_grid
mkdir -p /u01/sw_oracle chown oracle:oinstall /u01/sw_oracle chmod 777 /u01/sw_oracle
18. Authentication keys for root, grid, and oracle were created on all nodes in the environment.
ssh-keygen -t rsa
33 Dell SC Series Storage with Oracle ASM | CML1111
19. Copy root authentication key from node 1 to other nodes in the environment.
scp .ssh/id_rsa.pub
20. Copy root authentication keys from node 2 to other nodes in the environment.
scp .ssh/id_rsa.pub
21. Copy root authentication keys from node 3 to other nodes in the environment.
scp .ssh/id_rsa.pub
34 Dell SC Series Storage with Oracle ASM | CML1111
22. Copy the root authentication keys from node 4 to other nodes in the environment.
scp .ssh/id_rsa.pub
23. Add the keys to the authorized keys file on node 1.
chmod -R 0700 ~/.ssh cat .ssh/id_rsa.pub > .ssh/authorized_keys cat .ssh/
24. Add the keys to the authorized keys file on node 2.
chmod -R 0700 ~/.ssh cat .ssh/id_rsa.pub > .ssh/authorized_keys cat .ssh/
25. Add the keys to the authorized keys file on node 3.
chmod -R 0700 ~/.ssh cat .ssh/id_rsa.pub > .ssh/authorized_keys cat .ssh/
26. Add the keys to the authorized keys file on node 4.
chmod -R 0700 ~/.ssh cat .ssh/id_rsa.pub > .ssh/authorized_keys cat .ssh/
27. Repeat this process for the grid and oracle users. The process will be similar but commands will need to be modified. 28. Download the Grid and Oracle software into the staging area defined above.
35 Dell SC Series Storage with Oracle ASM | CML1111
linuxamd64_12102_database_1of2.zip linuxamd64_12102_database_2of2.zip linuxamd64_12102_grid_1of2.zip linuxamd64_12102_grid_2of2.zip
29. Download the necessary patch to mitigate the issue of creating ASM spfiles on 4K block disk devices in 12cR1 and place it in the staging area defined above. For information on 4K issues with Oracle, see documents located on MOS.
p16870214_121020_Linux-x86-64.zip
30. Change the privileges and ownership of the staging directories and zip files.
chmod -R 666 /u01/sw_oracle chmod -R 666 /u01/sw_grid chmod 777 /u01/sw_oracle chmod 777 /u01/sw_grid
chown oracle:oinstall /u01/sw_oracle/*.zip chown grid:oinstall /u01/sw_grid/*.zip
31. Unzip the software.
su - oracle cd /u01/sw_oracle unzip linuxamd64_12102_database_1of2.zip unzip linuxamd64_12102_database_2of2.zip exit
su - grid cd /u01/sw_grid unzip linuxamd64_12102_grid_1of2.zip unzip linuxamd64_12102_grid_2of2.zip exit
32. Create the ASM partitioned disks on the primary node.
/etc/init.d/oracleasm createdisk crs1 /dev/mapper/crs1p1 /etc/init.d/oracleasm createdisk data1 /dev/mapper/asm-data1p1 /etc/init.d/oracleasm createdisk data2 /dev/mapper/asm-data2p1 /etc/init.d/oracleasm createdisk fra1 /dev/mapper/asm-fra1p1 /etc/init.d/oracleasm createdisk fra2 /dev/mapper/asm-fra2p1 /etc/init.d/oracleasm listdisks ls -ltr /dev/oracleasm/disks
33. On the other nodes, scan for the newly created ASM disks.
/etc/init.d/oracleasm scandisks ls -ltr /dev/oracleasm/disks
36 Dell SC Series Storage with Oracle ASM | CML1111
34. Install the Grid cluster rpm that allows the cluster verification utility to discover shared disks.
cd /u01/sw_grid/grid/rpm rpm -ihv cvuqdisk-1.0.9-1.rpm
35. Create a staging area for the most recent version of cluvfy.
mkdir /u01/sw_grid/grid/cvupack chown grid:oinstall /u01/sw_grid/grid/cvupack chmod 777 /u01/sw_grid/grid/cvupack
36. Download the most recent version of cluvfy and place it in the staging area above then unzip it as user grid.
su - grid cd /u01/sw_grid/grid/cvupack/ unzip cvupack_Linux_x86_64.zip cd bin
37. Verify the Grid and Oracle configuration using cluvfy.
./cluvfy stage \ -pre crsinst \ -flex -hub
At the time this document was published, there was an outstanding issue with cluvfy that caused it to produce a false-positive on not being able to check all ASM disk devices. For additional information, see MOS notes:
Device Checks for ASM Fails with PRVF-5150: Path ORCL: is not a valid path [ID 1210863.1] Grid installation reports error on "Prerequisite checks" PRVF -5150 [ID 1498115.1] Bug 14048512 - PRVF-5150 : Path ORCL:DISK1 is not a valid path on all nodes [ID 14048512.8] GI 11.2.0.3 Installation On Errors PRVF-5150 saying ASMLIB path is not a valid path [ID 1474961.1]
1. After the environment meets the minimum requirements, proceed to install Grid and Oracle.
xhost + ssh root@
37 Dell SC Series Storage with Oracle ASM | CML1111
./runInstaller
a. In the lab environment a Standard cluster and not a Flex cluster was installed.
Note: It is not necessary to install a Flex cluster if Flex ASM is used.
b. Oracle Universal Installer (OUI) will eventually prompt for the network interfaces. Make sure that if a private interface for ASM traffic was created, that it is selected. The below example lists a public interface, and a shared interface between Flex ASM and the RAC private interconnect.
38 Dell SC Series Storage with Oracle ASM | CML1111
c. Select Use Oracle Flex ASM for storage.
d. Continue and complete the rest of the installation. After the installation completes, verify that Flex ASM was installed.
39 Dell SC Series Storage with Oracle ASM | CML1111
2. Verify the correctness of the environment.
su - grid
export PATH=$PATH:/u01/app/12.1.0/grid_1/bin export ORACLE_HOME=/u01/app/12.1.0/grid_1 export ORACLE_SID=+ASM1
# Flex ASM should be enabled asmcmd showclustermode ASM cluster : Flex mode enabled
# Cluster should be a standard (Normal) cluster asmcmd showclusterstate Normal
3. Login to each node and verify that they were created as a Hub node and that they are active, and that the Flex ASM cardinality is set to three.
su - grid olsnodes -n -s -a
crsctl status res ora.asm -f | grep 'CARD' CARDINALITY=3 CARDINALITY_ID=0
4. Display which hub node ASM is running on, and if ASM is enabled/disabled on the hub.
srvctl status asm -detail ASM is running on
crsctl status res ora.asm NAME=ora.asm TYPE=ora.asm.type TARGET=ONLINE , ONLINE , ONLINE STATE=ONLINE on
crsctl status res ora.asm -t ------Name Target State Server State details ------Cluster Resources ------
40 Dell SC Series Storage with Oracle ASM | CML1111
ora.asm 1 ONLINE ONLINE
crsctl status res ora.asm -l NAME=ora.asm TYPE=ora.asm.type CARDINALITY_ID=1 DEGREE_ID=1 TARGET=ONLINE STATE=ONLINE on
5. Display the Oracle networks that were created during the OUI installation of Grid.
su – grid oifcfg getif em1 172.16.16.0 global public em2 10.0.0.0 global cluster_interconnect,asm
6. Display the configuration of the Oracle listeners.
crsctl status res -w "TYPE = ora.listener.type" NAME=ora.LISTENER.lsnr TYPE=ora.listener.type TARGET=ONLINE , ONLINE , ONLINE , ONLINE STATE=ONLINE on
crsctl status res -w "TYPE = ora.asm_listener.type" NAME=ora.ASMNET1LSNR_ASM.lsnr TYPE=ora.asm_listener.type TARGET=OFFLINE, ONLINE , ONLINE , ONLINE STATE=OFFLINE, ONLINE on
7. If the configuration of Grid meets expectations, Flex ASM will be running and configured. Therefore, proceed with the following. No additional configuration for Flex ASM will be necessary. 8. Create the ASM disk groups using .${GRID_HOME}/bin/asmca, 9. Install Oracle using ./u01/sw_oracle/database/runInstaller 10. Create the RAC database using .${ORACLE_HOME}/bin/dbca
41 Dell SC Series Storage with Oracle ASM | CML1111
D 512 byte vs 4KB byte sectors
Since the consumer-grade HDDs were first introduced in the early 1980s, 512 byte sectors were used. But as areal density requirements kept increasing on disks, there was a need to increase the sector size. This was discussed in a 1998 paper submitted by the National Storage Industry Consortium, and after that work began on increasing the bytes-per-sector. By 2011, all hard drive manufactures committed to ship new hard drive platforms that supported a new format called Advanced Format (AF).
Advanced Format uses storage surfaces more efficiently for large files, and allows for the integration of stronger error correction algorithms to maintain data integrity at higher storage densities. These disks would be presented to the OS with physical block size of 4K bytes.
Up to this point, Oracle only supported devices presented to the OS with 512 byte physical sectors. So with the advent of the 4K byte physical sectors, Oracle binaries had to change. Depending on the Oracle configuration, including versions of Oracle software, there are a couple issues using 4K byte physical sectors with Oracle and a couple options available to mitigate the issues. With respect to the 12c (12.1.0.2.0) environment used for this paper, the only issue discovered pertained to writing spfiles to disk devices that are presented to the OS with 4K byte physical sectors. This caused issues for the Grid installer when it tried to create its management repository database, and during the creation of a database using DBCA.
To mitigate any issue caused by 4K vs 512 byte physical sectors, Dell recommends that the following options be executed in order listed and that Oracle Support is contacted for assistance.
1. If ASMLib is used, it might be possible to configure /etc/sysconfig/oracleasm to direct the ASMLib driver to use the size of the logical block rather than the size of the physical block. The ASMLib directive isn’t supported in all version of Oracle so please check My Oracle Support if it can be used.
ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true
2. Oracle patches are available for download. For a complete list of issues and available patches pertaining to 4K byte sectors, perform a search on 4K vs 512 byte sectors in My Oracle Support. Some of the notes will make reference to specific patches that must be applied to the Oracle environment to use 4K byte sectors 3. There are some My Oracle Support notes and threads that even point to a workaround of making sure the Storage provider would present the logical and physical block size of volumes that reside on Advanced Format devices to be 512 bytes. If Oracle Support is unable to resolve issues related to using 4K byte blocks, contact Dell Copilot for assistance.
42 Dell SC Series Storage with Oracle ASM | CML1111
E Additional resources
For Copilot support of Dell SC Series products:
SC Series storage Customer Portal Email: [email protected] (non-emergency business hours) Phone: 866-EZ-STORE (866-397-8673) (United States only) Global online support
The Dell SC Series storage Customer Portal is an online portal for existing customers. A valid portal account is required. Once logged in, click Knowledge center.
Dell TechCenter is an online technical community for IT professionals and is a great resource to discover and learn about a wide range of technologies such as storage, servers, networking, software, and cloud management.
Support.dell.com is focused on meeting customer needs with proven services and support.
Referenced or recommended Dell publications:
Dell Storage Center with Red Hat Enterprise Linux (RHEL) 6x Best Practices: http://en.community.dell.com/techcenter/extras/m/white_papers/20437964 Oracle Best Practices on Compellent Storage Center http://www.dellstorage.com/WorkArea/DownloadAsset.aspx?id=3055 How to deploy Oracle 11gR2 on RHEL6/Oracle Linux 6: http://en.community.dell.com/techcenter/enterprise- solutions/m/oracle_db_gallery/20212999.aspx
Referenced or recommended Oracle publications:
Oracle Linux Installation Guide for Release 6: http://docs.oracle.com/cd/E37670_01/E41137/E41137.pdf Oracle Linux Administrator’s Guide for Release 6: http://docs.oracle.com/cd/E37670_01/E41138/E41138.pdf Oracle Grid Infrastructure, Installation Guide, 12c Release 1 (12.1) for Linux, E48914-16: https://docs.oracle.com/database/121/CWLIN/E48914-16.pdf ASM Enhancements in Oracle Database 12c: http://www.oracle.com/technetwork/articles/database/asm-odb12c-enhancements- 2206102.html Oracle Database 11gR2 Database Administration Database Performance Tuning Guide: https://docs.oracle.com/cd/E11882_01/server.112/e41573/iodesign.htm#PFGRF015 Oracle Database Storage Administrators Guide http://download.oracle.com/docs/cd/B28359_01/server.111/b31107.pdf
43 Dell SC Series Storage with Oracle ASM | CML1111
Oracle 11g Database Concepts: http://download.oracle.com/docs/cd/B28359_01/server.111/b28318/toc.htm Oracle Database Release Notes 11gR2 for Linux https://docs.oracle.com/cd/E11882_01/relnotes.112/e23558/toc.htm Oracle Database Installation Guide 11gR2 https://docs.oracle.com/cd/E11882_01/install.112/e47689/toc.htm Oracle Grid Infrastructure Installation Guide 11gR2 https://docs.oracle.com/cd/E11882_01/install.112/e41961/toc.htm Oracle Automatic Storage Management, Administrator’s Guide, 12c Release 1 (12.1) https://docs.oracle.com/database/121/OSTMG/E41058-11.pdf Grid installation reports error on "Prerequisite checks" PRVF -5150 [ID 1498115.1] Device Checks for ASM Fails with PRVF-5150: Path ORCL: is not a valid path [ID 1210863.1] Bug 14048512 - PRVF-5150 : Path ORCL:DISK1 is not a valid path on all nodes [ID 14048512.8] GI 11.2.0.3 Installation On Errors PRVF-5150 saying ASMLIB path is not a valid path [ID 1474961.1]
44 Dell SC Series Storage with Oracle ASM | CML1111