UCRLJC-120489 PREPRINT

Management Issues for High Performance Storage Systems

S. Louis R. Burris

This paper was prepared for submittal to the 14th EEE Symposium on Mass Storage Systems Monterey, CA September 11-14,1995

March 1995

This isa preprintofapaperintended forpublication ina journalorproceedings. Since changes may be made before publication, this preprint is made available with the understandingthat it will not be cited or reproduced without the permission of the author.

DISR3IBUTION OF THIS .

DISCLAIMER

Tbis document was prepared as an account of work spomorcd by an agency d tbc ziaitcd States Government. Neither the United Stata Covennncnt nor the University of Californianor any dthar employees, makes any warranty, express or impiicd, or assumes any legal liabiity or nsporrsibilit y forthe amnmcy,compl&ms, or uscfh ofany information, apparat~s,prodact,or process disdased, or represents thatits pw wouldnot infringeprivatelyowned rights. Referurn hercia toanyspedficcommcmid pmduds, process, or SCRicc by name, tradcmdc, mPnufactunr,or otberrtc, doesnotnecessarilyconstituteorimplyitsendorsemmt, rrcommcndafh,orfn*orisg by the United States Covenunetat or the UnivcrJity of Womb. The viers and opinions of authors expWherein do not ncccssd' y state or reacathosc dtbe united States Goven?mtnt or the univerSity of CdifomiP, and shall not bc used for advertisingor product endorsement purposes. DISCLAIMER

Portions of this document may be illegible in electronic image products. Images are produced from the best available original document. Issues for High Performance Storage Systems

Steve Louis

Lawrence Livermore National Laboratory Livermore, California

Randy Burris

Oak Ridge National Laboratory Oak Ridge, Tennessee

This push to provide better management has Abstract generated ;111evolutionary advancement from purely reactive approaches to more proactive ones, sometimes Managing distributed high-performance storage utilizing knowledge-based expert systems. Today's systems is complex and, although sharing common organizationsare also in the midst of a radical change in ground with traditional network and , their computing and information infrastructures, from presents unique storage-related issues. Integration older, mainframe-based, highly centralized resources to technologies and frameworks exist to help manage new network-based, open distributed systems with many distributed network and system environments. Industry- management perspectives. These trends lead to a new driven consortia provide open forums where vendors and enterprise-wide view of management. users cooperate to leverage solutions. But these new approaches to open management fall short addressing the An enterprise-wide view needs of scalable, distributed storage. We discuss the motivation and requirements for storage system Initially, management applications and tools focused management (SSM) capabilities and describe how SSM on networks and components on networks, such as hubs, manages distributed servers and storage resource objects in bridges and routers. Next came management of host the High Performance Storage System (HPSS), a new systems, predominantly mainframes. Management of storage facility for data-intensive applications and large- applications running on mainframe systems followed. The scale computing. Modern storage systems, such as HPSS, subsequent transition to more open heterogeneous require many SSM capabilities, including server and environments led to distributed applications consisting of resource configuration , performance monitoring, multiple cooperating processes running on several nodes. quality of service, flexible policies, file migration, file The implementation of such distributed technologies is repacking, , and quotas. We present results of difficult, and managing them is daunting. initial HPSS SSM development, including design Formerly commonplace management activities such decisions and implementation uade-offs. We conclude with as event handling, centralized logging, perfoxmance plans for follow-on work and provide storage-related monitoring, security, accounting, configuration control, recommendations for vendors and srandards groups seeking and workload scheduling must now be handled across an enterprise-widemanagement solutions. entire enterprise using a highly distributed infrastructure. Increased user autonomy in these environments translates The management problem to loss of control over security, integrity, and backup, and to continuous interruption in the management Effective management of new information environment New open system choices introduced a technologies, whether for high-performance computers, for combinatorial increase in management complexity and the network, for the desktop, or for storage, cuts to the load-balancing problems. Consistent management tools heart of today's corporate concerns. Rapid technology began to be a necessity. Newer management solutions change, decreased budgets, and incmsed demand for tend to be non-proprietary, use open protocols, and . reliability present new management challenges in all fields provide application programming interfaces (APIs). of Computing. Hardware and software technology vendors Proprietary management solutions that cannot be easily must now provide efficient management solutions integrated into new open systems will be inadequate for alongside their products to help meet these challenges. rapidly changing environments.

DISTRIBUTION According to many vendors, integration across the should make everything look like a network, and manage enterprise is the new Holy Grail of management and the them with existing network management tools. However, solution to growing complexity. Integration across for real success, there should be a shift from managing multiple protocols, multiple operating systems, multiple networks and systems to management of the appiications networks and multiple applications is the overarching that run on those networks and systems. In the next goal. New technologies are required. Older, proprietary section, we describe some Current and emerging management platforms are slowly being replaced by newer management technologies and standards that may help open system technology directions, but open management realiie this shift. platforms do not solve all integration problems. Diverse management applications must still be integrated with Competing technologies these open platforms and there are many options for integrating them (e.g., process launching, alarm trapping, Understanding current open management command line interfaces, and APIs). technologies takes effort. There are management platforms, management tools, management toolkits, Complexity concerns management models, and management frameworks. Some technologies are viewed as of--the-sheffcommodity items. Integration may be the goal, but a looming problem Others are not yet well formed Still others are in open (and hidden Achilles Heel of open distributed computing) conflict. is system scalability and corresponding management complexity. Networks have grown from small 10 megabit Management platforms Ethernet networks to very large data pipes and switches. Unintelligent network nodes have become smart. Smart Since network management is viewed as the most network nodes have expanded to themselves become entire evolved technology and closest to a , we srart networks. Management of networks (and some systems) there. Several network management platfoxms exist. Most used to be straightfoxward using the Internetdeveloped are based on UNIX with Hewlea-PacM Openview, IBM Simple Network Management Protocol (SNMpV1) 111. NetView for AIX, and SunSoft Encompass as notable Because of scalability concerns, there are now several market leaders. Other vendors, such as Cabletron, follow-on activities suggesting how to expand SNMPvl's NetLabs, Openvision, and Peer Networks provide ability to handle thousands of management agents, sub- platfom-related supporting technologies. Platforms agents, and Management Information Bases (MIBs), and to provide basic management services, such as easy-to-use monitor and control millions of managed objects. graphical user interfaces (Guls), resolace discovery and Unfortunately, many of these newer activities lead to mapping capabilities, data browsers and editors, alarm vendor-proprietary schemes and protocols. capability, and protocol support. They differ in their Other approaches to controlling complexity stress senice implementation methods and &van& services more automation and less manual intervention. Many offered. Advanced seMces include alarm cone la ti^ vendors suggest adopting a manugement-by-exception distributed management capabilities, resource modeling features, and support for non- Protocol (non-IP) philosophy instead of more common polling procedures. Event- and alarmdented management employs communications. automated limitatim of management platforms is that they setting of alarm thresholds, alarm filtering criteria, and A alarm correlation to help reduce complexity. Correlation are not much more than enhanced IP network monitors. Platforms primarily act collectors without means that multiple alarms are examined and analyzed by as data a (possibly automated) process for clues about what has capabilities to do much with those data. They provide an infrasaucture for other management tools, but not happened and what approPriate responses might be. Another common technique is drill-down. This is a way to necessarily a comprehensive solution. Applications are systematically examine more detailed laym of still needed to meet specific needs. Most platforms also management information, thus preventing drowning in a lean heavily toward networking issues at the expense of of supeffluous misleading Most modem distributed systems, applications, and integrated enterpriSe flood ar data. management. management tools provide drill-down capabilities, but not necessarily in consistent ways. Management models The bottom line Models for management, especially in networks, can Inattention to management results in be intrusive or nun-intrusive. In an intrusive model, detail management processes simply out and objects to ineffective and inefficient system design. Lack of go kick appropriate management function places a huge burden on see if they are still alive. This works well for non- users and support personnel to cope with new problems. intelligent devices, but can impact perfoxmane of . communication networks if polling activities are Almost everything in a system can be logged or transformed blinking icons, but what does a manager excessive. In a non-intrusive model, objects have abilities into to tell a management process when something of interest really want to see and do? There are those who believe we is happening. This kind of model works well for enhanced interhce schemes for SNMP are described in computers and intelligent equipment. Non-intrusive various Internet RFCs [2,31. SNMh.2 [41 addresses models usually have less impact on network performance. several of the shortcomings in SNMPvl, including Recently, systems management extensions have security [5]. been developed for network management models and Another network management model that has been network management protocols. Typical system and extended to include systems and applications concepts is application functions are automated scheduling, backup the IS0 OS1 management model [6,7]. The OS1 model, and restore, tape management, , while richer and more complex than SNMP, has not found inventory management, print management, performance acceptance in the form of widespread use in the United management, software distribution, and user/group states. management. However, introducing system and One area that remains consistent through most application extensions to standard networking models management models, whetber for networks, systems, or raises new issues. applications, is the use of objects to encapsulate the The original SNMP protocol supports simple GJ~T behavior of manageable entities within a system. An and SET attribute operations and has a trap facility for object can be viewed as a set of state values and operations alarm handling. It relies on a single SNMP port for each that together represent object behavior in response to system. SNMP agents listen on that single UDP port for requests. The state values of an object are sometimes management requests. There are severe limitations using a called the object's atuibutes. The object operations are monolithic agent for all possible combinations of system normally accessible through well-defined interfaces. A and application components in a complex environment, managed object is an object to which a management hence the recent activity in new sub-agent protocols. policy applies and whose behavior can be monitored Limited security in SNMPvl is also considered a major andor changed by amanagef (see Figure 1). Managed problem for managing systems and applications. objects can represent hardware, software, data structlrres, or Potential improvements to SNMP have been non-computational entities such as people. explored to address limitations in its original design and inherent network outlook. Agent multiplexing and

Event Notifications

Control Request I------Response Information Request

Figure 1. Generalized object model for management applications.

community that developed believes it is more Standards and consortia helpful to consider standardsSNMP based upon merit, not pedigree and that there important distinctions Management integration appmaches exist in the between dejure[SI, aud defactoare standards. The fmer form of several standards, frameworks, and industry are legitimate by force of law, but the latter become consortia. Some of these efforts have found wide legitimate by popular acceptance. De jure standards can, acepmce, others hold promise for yet-@be-realized and do, fail to gain popular use. Usable standards, whether capabilities, and few have have a stalled or been disbanded. de jure or de facto, require multiple vendor involvement Many times there is little coordination between these and convergence. Motivation to support a standard may be 0 approaches, forcing vendors to track multiple efforts and driven by several competing factors, including finding a confusing users about what standards are important. profitable extension traditional markets, selling more of The (Internet Engineering Force) to IETF Task an existing product, or winning a related war with competitors. Open Distributing Rocessing (ODP) is also a new "When is a standard really a standard?" The following effort attempting to solve problems of software interaction anecdote, taken from an article on the development of a by proposing a common framework for distributed new SNMP MIB 191. illustrates the difficulty of vendor systems. This framework, called the Reference Model for convergence in the standards process: Open Distributed Processing (RM-ODP),is now an international standard 11 11 built upon several years of "Anyone who has been in the computing industry distributed computing reseaxh and experience with awhile is all too familiar with the way that standards commercial technologies like CORBA and DCE. It sometimes go when they go bad (we'll use the term describes an architecture within which support of Widger as the hypothetical technology being standardized): distribution, interworking, interoperability and portability "Vendors can each develop their own Widgets, and can be integrated. CORBA, WE,and RM-ODP only then try to make them de facto standards. This may result peripherally address management issues, but are important ' in one winner, but usually leaves a group of die-hards who because they shape the distributed environments that cannot embrace it, for one reason or another, and who enterprise management solutions must support. continue to cling to their own Widgets. SNMP was recently used to standardize management "A group of vendors can form the Open Widget for relational database management systems. Leading Consortium, defining a standard that everyone is then RDBMS vendors, together with users, consultants, and supposed to adopt. Usually, though, the Consortium IETF members produced a standard SNMP MIB for excludes one or more key vendors, who then form the managing an RDBMS [ 121. This activity is an example of rival Integrated Distributed Widget Group. Dueling press a new activity that fulfilled its charter in less than six releases follow. months, providing a standard before any proprietary "A formal standards body takes up the process of solution became entrenched. The standard addresses basic Widget standardization. Depending on how the standards functionality and was developed to discover information body functions, this may take months to complete, or about all brands of DBMSs, such as vendor identity, years, or it may never complete at all, eventually being name, and administrator, and to characterize (by size) the eclipsed by newer Widget technology. Indeed, sometimes number users and activity levels. Rivate vendordeveloped this gridlock may serve the interests of one or more of the MIBs are dowed for application-s@ic drilldown parties, who don't really want a standard to be adopted at inquiries. all. Newer management models that integrate existing SNMP standards, databases, file systems, storage Examples of less-than-successfulopen distributed facilities, and user application management have begun to management efforts abound (e.g., UI Atlas, OSF DME, emerge. Some models suggest using RPC technologies COSE systems management). Nevertheless, the standards rather than SNMP (see Figure 2). RPCs are used to process continues. Recent activities (standards and communicate between enterprise components and a otherwise) that may have an impact on future storage systems management platform running enterprise-wide systems and management architectms are described management applications. However, RPC-based models below. can also have scalability problems, such as resource discovery issues when tens of thousands of agents, MIBs, Recent developments and objects are present in distributed environments. The Management Integration Consortium (MIC) is a Storage system problems group recently formed by vendors to define a common management data model and management data access Traditional goals for network and system mechanism [lo]. This effort is more a working forum for management are focused toward minimizing seMe vendors xather than an official standards body. Stated goals downtime and maximizing service availability, reliability, of MIC are to develop a standardized dambase schema and and security. Recently, other goals have appeared, APIs that facilitate data exchange and sharing between including lower administrative overhead and support of management applications. One motivation of MIC is to overall objectives. All these goals also apply to have third-party vendors drive the schema and API storage services. While several technologies exist to help definition instead of major platform vendors. MIC networks and systems attain their management goals, believes that off& standatds bodies are not wellequipped well-focused storage system management (SSM) to deliver simplicity, and that platform vendors may have applications and tools are hard to 6nd. more incentive to deliver differentiation than integration. Due to the lack of adequate SSM tools, many While not targeted toward management, the Object storage systems lack a coherent approach to management, Management Group's Common Object Request Broker sometimes providing an eclectic mix of management Architecture (CORBA) and the OSFs Distributed function based on local site requirements and peculiarities Computing Environment WE)are examples of existing of system implementation. In the high-performance technologies that address heterogeneity and interworking storage community, storage systems were historically in large, distributed computing infrastructures. developed in-house and not purchased from commercial ) suppliers. The high-performance storage system ran an enterprise was not well-recognized. independent entity. Thus, the need for compatibilityas or integntion with other management applications in the

~ Management Applications Management Applications

SNMP Management Platform I RPC Management Platform I t SNMP 1 IMaster Agent I 101Application In1Application B IApplications J

Figure 2. SNMP and alternative RPC-based management models.

specific management roles and rules, but rather provides Reference models and frameworks an extensible structure in which to implement and enforce site-specific storage system policies and pmedures. One effort that examined SSM requirements was the Frameworks can provide flexibility, scalability and a IEEE's Storage System Standards Working Group foundation for rapidly responding to change. However, (SSSWG).Using the site management ideas of the frameworks will not solve specific problems. Instead, they original Mass Storage System Reference Model Version 4 enable and fadirate good solutions. Most existing (MSSRMv4) storage system vendors, researchers, 1131, frameworks use objects, as they are scalable, secure, developers, and users discussed what might constitute a reliable and suppcxt reuse. Unfortunately, there not yet standardii Eramework for functions of are SSM,speciaIized any widely accepted standards or frameworks for distributed SSM,and how SSM relates to other management applications management. technologies 1141. These views incorporate both managed High-perfommx distributed storage systems share objects and management by policy (see Figure 3). many of the management concerns of other distributed Progress toward standadm' tion is slow. A SSM applications, but also have important management issues difficulty faced by IEEE SSSWG was that many not necessarily found elsewhere. These storage issues participants were not vendors. Instead, they were must be addressed today, and solutions must be integrated motivated system developers and managers who faced the smoothly into future open distributed management reality of a continuing concern at their sites. SSM as own standards and technologies. Another difficulty is the existence of other management standardization activities that., while not specifically Storage issues addressing storage, are attempting to provide standards far f the management infrasauctlne needs of general distributed Management services are sometimes grouped into systems, thus complicating a scope definition for SSM functional categories such as fault, coi@uration, security, Standards. accounting, and perfmcemanagement. The OS1 . Astandar~frameworkfordistributedstorage management model uses this categorization. The IEEE system implementations and distributed storage system MSSRM adds transaction management. RM-ODP uses a management is needed. The IEEE MSSRM is a step in slightly different grouping deemed appropriate for ODP this direction. An SSM framework does not mandate systems - namely configuration, quality of service, accounting, performance, and policy definition. Security components that fail represent the potential loss of functions are not included in RM-ODP management, but irreplaceable, permanent data. Hence, fault management in management interactions are subject to policy control. storage includes comprehensive proccsdures for backup and Although functional groupings are similar for most recovery of critical data, both user data and system management services, the specific requirements of storage metadata. The integrity of the system metadata may also systems are quite different from management of need to be continuously maintained through integrated communications equipment or small systems. transaction management facilities. Unlike failing networks or systems that can be rerouted, reloaded, rebooted, or replaced, storage system

Service Request of P1244.m * 0 0 Service Request AnP P1244.xxxServersand PI244.m Objects 1 - Events

SSM ti Request 000 I------00 Storage System Management via Internal Policy I "Provider" Management Applications of P1244.m Storage Services

Figure 3. Conceptual view of EEE SSSWG storage system management

The most important issues for storage systems storage systems, but must be. capable of tracking up to involve hiemhical storage management (HSM) millions or billions of objects of vastly different type operations. Data caching and migration (movement of data (e.g., bitfiles, virtual volumes, physical volumes, and between different storage levels), and media repacking caruidges). (consolidation of data to reclaim unusable space) are Once resomare defined (ordiscovered) and common operations. Management of these operations configured to suit one's needs, there still needs to be constitute much of the day-Wy administrative and security or policy control over the use of configured opemtional procedures for high-performance storage, yet resources and management operations. Security €or some are not well addressed by cment network and system sites is paramount because stored data might be classified management tools. SSM requires an ability to define or sensitive. Security violations need to be detected and policies that ensure HSM operations meet specific site reported. Management might maintain logs of violations requirements. and storage system requests of security interest. Another issue for storage systems, especially those Because storage systems contain fixed resource that are highly distributed is flexible merand resource capacities (Le., finite allocable space on usable devices), a configuration control. This includes not only basic method of user control or coercion is necessary to prevent functions to start, stop, and display server compents, abuse of limited resources. Some systems employ an but also to find out what components exist, how to accounting process that charges for the resources remove them, and how to create them. Automatic consumed. An alternative to charging is to use quotas that discovery capabilities are commonly used in network aummtically prevent access to resources when a user (or management applications to display iconic views of active group) limit is reached. network components. Similar capabilities are needed for Quality of service (QoS) is a set of client-understa attributes, expressed in a client-understood language, requirements for modularity of software components and describing expected characteristics of a particular service. an integrated SSM capability. Each software component is QoS is used to help clients observe or influence the responsible for a welldefined set of storage objects, and collective behavior of distributed system resources. QoS acts as a service provider for those objects. Two parameters for storage might include minimum and specialized management components, the System Manager maximum file sizes, allowable YO operations, remote and the Dura Server, together comprise HPSS SSM, vaulting capability, data transfer rate, perceived delay, which monitors and controls the resources of the system. availability, reliability, and security. QoS in storage HPSS overview systems can be implemented as a fured or dynamic hierarchy of devices, each level in the hierarchy having a different combination of QoS characteristics. The NSL and HPSS are indusny and U.S. Cost-effective utilization of storage resources and Department of Energy organized to develop construction of proper policy needs adequate performance and commercialize new technologies for high-performance monitoring capabilities. This will require intelligent distributed storage [ 161. Principal developers for HPSS are the Department of Energy's Lawrence Livennore, definition and use of measurement data (metrics). Storage U.S. system designs should clearly define and Correctiy balance Los Alamos, Oak Ridge and Sandii National Laboratories, the level of instrumentation detail within the system's and IBM Government Systems. Other developers include components. should provided for throttling, Cornell University, NASA Lewis, and NASA Langley Tools be Research Centers. A motivation for HPSS to filtering, collecting, verifying, and correlating data There major was are no standards for storage-system-specifk metrics, develop a high-speed storage system providing scalability although Nopen's Working to meet demands of new high-perfoImance computer Group (PMWG)has defined several metrics for UNIX UO applications where vast amounts of data are generated, and to meet the needs of a national information infrastructure operations, and disk and tape devices [151. Management of large storage systems ~71. has HPSS a networkedcentered architecture 1181 and historically been perfomed by a dedicated operations staff has or by howledgeable systems programmers well-grounded its software components are based on IEEE MSSRMv5 in storage configuration,maintenance, and intervention (see Figure 4). The architectureincludes a high-speed network for transfer and a separate network for device procedures. As distributed processing and distributed data management place more demand on existing staff, control and management. The control and management operation and management of storage systems will likely network uses DCE RPC technology. Control and data fall to the those that manage networks and systems. This transfer networks may be physically separate or shared in presents problems if these new are not actual implementations [ 191. Another feature of HPSS is administrators support for both parallel and sequential inpudoutput familiar with storage system management issues. The (YO). need for storage systems to provide proactive management The UO architecture is designed to scale as technology capability and smoother integration with other improves by using data striping and multiple data movers a parallel management systems will be required. as YO mechanism. In summary, a well-constructed storage system HPSS was designed to support data transfers from of megabytes up to a gigabyte second. File should meet the following SSM concerns: hundreds per size scalability must meet the needs of billions of data Hierarchical storage management sets, some potentially terabytes in size, for total storage capacities in petabytes. The system must also scale Fault detection and data recovery Resource configuration control geographically to support distributed systems witb Secure management operations hierarchies of distinct storage systems. and usage limits components of HPSS were also designed to scale along Quality of service flexibility with the rest of the system. Balanced internal instrumentation Proactive/seE-managed components Management philosophy

We describe in the next section how these Concepts from current management standards and requirements were add~ssedin a modern, high- technologies were incorporated into the HPSS SSM performance, distributed storage system, in tbe presence or approach where feasible. Initial design began with abstract absence of suitable management technologies for storage. mauaged object definitions and was based on widely accepted management model cxmstructs. These definitions HPSS case study were subsequently relined after meeting with server development teams. This helped in developing an overall HPSS, a development effort of the National Storage approach to uniform software server development and a Laborato~~(NSL), is a new high-perfomance stomge flexible instrumentation process. HPSS is written in C, system for data-intensive applications and large-scale not an object-oriented language, but objects are used to computers. Key elements of HPSS design were represent resources and components throughout the system. This provided a good conceptual view of what yet agreed on comprehensive standards for management needed to be managed. We focused on administrative and APIs or protocols, we avoided using a specific protocol operational conuol of the system with the understanding such as SNMP for management communication between that those who would be exercising control were not the servers and the System Manager. We, instead, relied necessarily system software developers nor experts in on simple RPC-based management interactions, similar to HPSS storage intemals. other inter-server communication in HPSS. We believe The highest priorities for HPSS SSM were to that the SSM architecture currently in place is easily develop necessary management functionality. As a result, adaptable to what finally emerges as the protocot and HPSS SSM has a robust set of fault, configuration, integration strategy of choice for enterprise-wide security, accounting, QoS, HSM, and instrumentation management of distributed systems and applications. capabilities. Since system and software vendors have not

INFRASTRUCTURE /DCE and RPC Transaction Mgmt Metadata Mgmt Security Logging / Client(s) Manager (SSM)

! Physical Volume Repository (PVR) Data Movement / Figure 4. HPSS software architecture.

A decision was also made to avoid a reactive requirements for a quality of seMce capabiiity, SSM , where problems are only identified provides tools for administrative definition and after the system is in serious trouble. Instead, a proactive management of HPSS Class of Service (COS), a data management philosophy was used. SSM can query almost structure that helps identify performance, media, and usage any internal server information through the display of ataibutes of resource objects managed by the servers. managed object attributes. An ability to provide COS is used to ensure that HPSS meets varying client management by policy was also built into the SSM expectatioas €or service quality [u)].These design. Policies in HPSS are implemented as metadata administratively managed COS definitions are internally that can bemodified by administrators and operators to mapped to lower-level storage class attributes more easily suit the specific needs of a site. For example, migration understood by the servers that perfom actual resource and repacking of fdes are conmiled through identifiable management. ntis helps isoiate server developers from policies associated with HPSS service and storage classes. external management processes unrelated to basic server Policies are also used for HPSS accounting and security functionality. Similarly, adminismtive filtering of log management. Servers are written such that management messages, events, and alarms sent by servers is performed decisis are not hardcoded into software but can, instead external to servers (it is perfamed by a logging seMe be determined from policy files or from management infrastructure). Comparable approaches to shielding . requeStS. application developers from complex management Administrative management ccmcems were shielded concerns have been used by others [Zll. from servers where possible. For example, to satisfy The Data Server receives information from the System Data structures Manager and formats it for display to the GUI through graphical display runtime @Is. The Data Server also To properly manage distributed HPSS servers and receives operator requests and converts them to a form the resources, configuration and state data describing the Manager can use. Any Data Server accesses to Encina are servers must be consistent. To meet this goal, necessary made on the Data Server's behalf by the System Manager. managed objects and metadata were defineQ together with Operations provided by the System Manager to the interfaces for object manipulation and communication. Data Server include starting, stopping, modifying Entities in HPSS are of two types: (1) servers, and and states of servers, configuration of Encina files, importing (2) resources managed by servers. Servers axe described by and exporting physical media, Creation of new virtual two types of managed object: (1) common, and (2) volumes, control of storage devices and jobs, viewing and specific. The common server managed object contains updating managed objects, and delogging. In standardized server information such state Its SNMP as data. terms, the System Manager primarily acts as an structure is generic and shared by all servers managed by intelligent master agent for the system. Servers Most servers are also by an additional HPSS SSM. described may be viewed as subagents communicating management specific server managed object containing information information through the System Manager. relevant only to that server. In a more object-oriented The Data Server (or other client in a Data Server system implementation, the specific server information role) makes contact with the System Manager with a could be used to define a new server subclass that inherits Checkln function. The System Manager retums a unique all generic server information Erom a common server identifier for the Data Server client and several lists that superclass. MISwere implemented to allow SSM to get known servers, drives, and service classes. All and set attributes for each of server managed object. describe type alarms, events, and notifications issued to SSM by other There are also several managed objects containing servers (through the logging service) are received by the information about non-server HPSS entities representing System Manager forwarded to the Data Server logical and physical such drives, camidges, and as resources, as appropriate. Requirements for filtering of log data are met virtual volumes, physical volumes, storage segments, and through administrative definition of logging policies. bitfiles. Attributes of these managed objects include One of SSMs major goals is ease of use for the information useful in determining overall system status administratorhpemtor. To achieve this, we implemented and perfmnance. APIs to get and set attributes of these management functions through a GUI with a managed objects were also implemented. all standar- look-and-feel. Effective layout of a user To satisfy the requirement for flexible resource interface can tedious since each must codiguration, server and some non-server entities be datum be each interpreted and placed on the screen, so a commercial (e&, cartridges and drives) have configuration-specific layout tool, Sammi, from Kinesix, was chosen. Sammi information associated with them. This configuration was also used for storage system management in NSL- information is stored as metadata under the control of an UniTree, a previous system developed at the NSL [231. internal Metadata Manager. The Metadata Manager, As a predecessor to HPSS, NSL-UniTree shares based HPSSTransarc's kinaproduct, also helps provide on some HPSS high-level SSM design. Both systems use a transaction management capabilities in HPSS [22].MIS general Health and Status display and both separate the to create, read, write, update, and delete conf&uration Server and System Manager functions. However, the metadata are available to SSM the other servers, Data and to NSL-UniTree GUI is more lited than HPSS and does thus ensuring the consistency of system metadata. not use DCE for communication or Encina for metadata Simii to the two managed objects representing management. There no managed objects in servers, there are two categories of contiguration metadaw are actual NSL-UniTm and the operator intertiice was not designed (1) common, and (2) specific. The variables contained in to the sole mechanism control of the system. common metadata include items like server name, a be for universal unique idenwier, and node and security SSM operation and information flow information. Specifc metadata are unique to each mer. Some HPSS servers require notification whenever their Many SSM operations, such as starting and configuration metadata is changed. Other servers prevent stopping serva are typical of Other distributed SSM from changing configuration data while the server is application management tasks. ?bese basic SSM running. operations axe good candidates for an enterprise- management integration strategy, as they should be easily design portable to existing management platfom technologies. Below is a stoplstart server example. There are three primary elements to the HPSS SSM Assume the system is already running and a new design: the GUI (graphical displays), the operations Data operator has arrived. The operator must fvst log into the Server, and the System Manager (see Figure 5). The system utilizing a logon screen and security features that System Manager communicates with all HPSS servers to prevent unauthorized access to privileged management acquire state information and to relay operator commands. operations. All SSM operations have an internal security level associated with them. This helps meet the operator can open those windows at any time and scroll requirement for administrative policy control over through them. If the desired information is unavailable, management operations. Having logged on, the operator is the operator can also perform a dehg operation -- dump an presented with a HeaIth and Status Display presenting active or archived log file containing information that may overall status information with and an aggregated not have been routed to the SSM servers. Delogging system state represented in both colors and words. In operations can filter log messages on server type, time, addition, this window has icons representing the alarm severity, and other criteria. aggregated operational state of HPSS devices, servers, and After studying all available information, the operator storage entities. has determined that the sick server must be killed and Let us now suppose that the server icon shows red; restarted The operator can either tell the server to stop something is wrong. The operator can perform a drill- through changing a field in the HPSS Server Managed down operation by clicking on that icon to get a sick list Object, or can make a menu selection that will present a containing a list of servers that have problems. When the list of servers from which to select the one(s) to stop. operator clicks on one of the servers in the list (another After the server has been stopped the operator can start it drilldown operation), a new window will be presented again through a menu selection process. When the server containing the common and specific managed objects for comes back up, the operator can make another menu that server, together containing all the relevant selection to display the managed objects representing that information maintained for that server. server to ensure the server is functioning properly. The The operator also has historical information flow of information through HPSS when starting a server available in the form of alarm and event messages. These is briefly described below. are contained in separate windows accessible through a menu bar at the top of the Health and Status display. The

Sammi Runtime Environment

Graphical Displays

f DCERPCs f

System Manager DCE RPCs I HPSS System Management Node I HPSS Metadata Manager I I I

Figure 5. HPSS storage system management.

First, the command is fielded by the Sammi run- The System Manager establishes communication time system which presents it to the Data Server. The with the newly started server and retrieves the managed Data Server uses internal information it maintains to objects, which it then passes to the Data Server. The Data formulate a request to the System Manager. The System Server uses smsinformarion contained in the managed Manager instructs the Startup Daemon (a special process objects to determine overall system status and to remove on each HPSS node) to start that server, and tells it where the now-healthy merfrom the sick list. When the the common and specific configuration information can be operam asks to see the managed objects, the Data Server found. Once the semer starts, it reads its required fields the request adinstructs the window mgerand configuration information from the Encina-maintained Sammi to open and fill in the windows. metadata The SSM GUI and menu system provide additional management capabilities such as system configuration, those under developmen a MIC would ease this pmss. alarm and event filtering, cartridge impcrt, and creation of During final design, coding, and integration physical and virtual volumes. Sometimes these storage test, it was necessarySSM to identify and remedy all remaining system operations are complex. Defining new cartridges inconsistencies, clarify SSM functionality, and revise means defining the Storage Server code. Fortunately, the design of the system, particularly data structures for the cartridges, which include physical the set of MISand managed objects, remained highly volumes, virtual volumes, and storage segment maps. stable through the server integration process. This , When a chain of related operations like this must take stability was, in part, the result of careful initial design place, an attempt is made to make the administrative work, and was to be of help developing the final SSM interface as simple as possible. Unlike the starting and implementation. stopping of servers, a relatively common distributed In tbe design of HPSS Version 2, the developers application operation, management operations for the used the lessons from Version 1 work. Because of more defmition of new cartridges and related storage resources familiarity with SSM and geographically distributed will not likely be standardized by current management collaborations, server writers were able to give earlier and standards activities. continuing consideration to the way their server should be managed. Implementation experiences Conclusions The HPSS project is a collaborative effort and represented a new "disaibuted development" approach for There are similarities and differences in management the participants. The geographic dispersion of the of networks, systems, applications, and high-performance developers, and the lack of cleat standards to guide the storage. Storage systems present management SSM implementation process, mated some unique requirements not necessarily addressed by existing experiences. management technologies. When similarities exist, we Not all development teams progressed through believe it is sensible to leverage previous work or existing design, object definition, implementation, and test at the technology. For unique characteristics of high-performance same rate. As a consequence, some teams produced early storage it may be necessary to develop new function. prototypes for SSM functionality, and some teams In HPSS, we used existing approaches, including produced later ones. These SSM capabilities were not managed objects, GUIs, RPCs, and transaction necessarily identical &om server to server because of management where practical, and developed our own requirement and design changes as the overall mechanisms where necessary. HPSS provides policy implementation progressed Thexe were differences in management of HSM migration and repack operations, routine name, status reporting, and alarm and event system metadata integrity through an integrated Metadata definition. These issues were successfully resolved during Manager, flexible resource configuration, event and alarm final implementation and system integration. capabilities, and authorization control for administrative Several implementation issues concerning more Operations. HPSS also provides service quality complex administrative tasks in HPSS were not decided management through its Class of Service data structures, until late in the development process. For example, some and thorough server instrumentation for current and future objects in HPSS are shared between sewers. This led to measurement, monitoring, and proactive self-management. several issues. Which server really owns the object? How Version 1 of HPSS underwent final integration test should the managed object@)be structured? What does an in early 1995 and was installed at several developer sites. operator have to know about the details of the sharing? A hybrid version (incorporating Version 1 and new disk Who should do the translation between the object an support features from Version 2) was successfully administrator can view and the possibly multiple objects demonstrated at Supercomputing '94. HPSS Version 2 is comprising the resource in HPSS metadata? scheduled for release in late 1995. It will include several SSM must communicate with all other servers, but new SSM features lacking in Version 1, including new it is not an infrastructure service (as are other HPSS support for disk resources, migration, purge, repack, and routines or tasks that must commuNcate with all servers). accounting operations. It is instead more of a superstructure, and it is in SSM There are SSM areas that we would like to improve that all information from all servers comes together. It is upon in future releases. These include a more easily SSM that needs to synthesize the information to produce managed classsf-service capability, better ways to utilize new information (such as an overall system status). As a policydxiven management, easier control of devices that consequence, SSM is one of the places where are shared between semen, more powerful accounting inconsistencies in use of common data or functions (including quotas), better performance monitoring, and become visible. This is comparable to the enterprise resou~cediscovery capabilities. Some areas targeted far management problem where several distributed future extension are Wig pursued in system and applications must all communicate management applications management areas, so it is likely that information in a consistent way to a central management emerging technologies can be leveraged for use in HPSS. platfm. Standard management data structures such as One step toward an enterprise integration strategy might be to introduce SNMP agent or sub-agent code into the HPSS servers to allow simple management operations to References be controlled by existing network management platfoms using SNMP protocols. Several vendors offer such agent 1. J. Case, M. Fedor, M. Schofstall and C. Davin, code for insertion into non-SNMP or other legacy Simple Network Management Protocol (SNMP, applications. RFC 1157, May 1990. Because high-performance distributed storage is a small market (compared to more widespread backup and 2. M. Rose, SNMP MZJX Protocol and MIB, RFC restore applications), gathering a critical mass of vendors 1227, May 1991. to produce workable standards in this is difficult. area This 3. G. Carpenter and B. Wijnen, SNMP-DPI Simple is evident for components at the client end of the IEEE Mass Storage System Reference Model, where dividing Network Management Protocol Distributed Program lines between storage, file systems, operating systems, Interface, RFC 1228, May 1991. and applications become fuzzy. An obstacle for specific storage system management standards is the large push for 4. J. Case, K. McCloghrie, M. Rose and S. open management standards, integration solutions, open Waldbusser, Introduction to version 2 of the Intemet- frameworks, and convergence in overlapping computing standard Network Management Framework, RFC 1441, April 1993. areas. Isolating storage-system-specificmanagement topics from general management is hard, but nevertheless, should still addressed. 5. J. Galvin and K. McCloghrie, Security Protocols for be version 2 of the Simple Network Management A suggested approach for successors to the IEEE SSSWG (or any groups interested in SSM) is to follow Protocol (SNMPv2). WC1446, April 1993. the model used in creation of the RDBMS MIB. was This 6. ISOAEC 1OO40, Information TechnologyAIpen not a true de jure standards effort, but due to its well- Systems Interconnection-S ystems Management defined scope,perceived need, and vendor acceptance was Overview, 1991. successful within reasonable time frames. Until SNMP agents and managers disappear, (unlikely in the it U.S.), 7. JSOiIEC 10165-1, Information Technology-Open may be reasonable to develop an overall storage system Systems Interconnection-Stture of Management MIB. MIBs for specific storage devices have already been Information, Part 1: Management Information developed by individual vendors, but an overall storage Model, 1991. system MIB has not been attempted. The impetus behind enterprise-wide management and 8. D. Crocker, the Way, The standardized approaches is gaining momentum. There Standards IETF are Interoperability Report, Vol. 7, No. 9, September still storage-system-specific issues for large, distributed 1993. environments. We trust that the individuals, vendors, consortia, and formal standards bodies investigating 9. R. Purvy, Using SNMP to Manage Relational management technologies and storage system applications Databases Conference Proceedings of the Enterprise will consider bow to bring high-performance storage Management Summit, San Jose, CA, November 14- management issues, such as those addressed in HPSS, 18, 1994. into the mainstream. The result will be useful management tools and applications for high-performance 10. E. Olinger, ed., Open Repository Specification - distributed storage and, consequently, much less need for MIC0001: Meta-Schema Specification, Management highend developers to continue down an independent path. Integration Consortium Draft, February 1995. Acknowledgments 11. ISOiIEC JTCl/SC21WG7, Reference Model for Open Distributed Recessing (RM-ODP), ITU-TS This work was, in part, performed by the Lawrence RK. X.901-904 I ISOlIEC 10765-1, -2, -3, -4, Livermore National Laboramy and the Oak Ridge 1994. National Laboratory under the auspices of U.S. Department of Energy Research and 12. D. Brower, R Purvy, A. Daniel, M. Sinykin and J. Development Agreements, and by IBM Government Smith, Relational Database Management System Systems under Independent Research and Development and (RDBMS) Management Information Base (MIB) other internal funding. We wish to acknowledge the using SMIv2, RFC 1697, August 1994. contributions of Vicky White and Dan Million of Oak Ridge for their work on the HPSS System Manager and 13. S .Coleman and S. Miller, eds., Mass Storage Data Server. We also wish to acknowledge Tyce McLarty System Reference Model, Version 4, IEEE Technical of Los Alamos, and Tracy Tran and John Nguyen of IBM Committee on Mass Storage Systems and Government Systems for early work on SSM design. Technology, May 1990.

This work was performed under the auspices of the U.S. Dept. of Energy at LLNL under contract no. W-7405-Eng48. 14. R. Garrison, et al. (eds.),Reference Model for Open Storage Systems Interconnection: Mass Storage Reference Model Version 5, IEEE Storage System Standards Working Group (P1244), September 1994. 15. L. Traister, ed., An Introduction to the UMA (Universal Measurement Architecture) Specifications, Version 1.O, Performance Management Work Group (PMWG),NOpen, April 1994. 16. R. Coyne, and R Watson, The National Storage Laboratory: Overview and Status, Roc. Thinth IEEE Symposium on Mass Storage Systems, hnecy, France, June 13-16,1994. 17. R. Coyne, H. Hulen, and R. Watson, The High Performance Storage System, Proc. Supercomputing '93, Portland, OR, November 15-19,1993. 18. D. Teaff, R. Coyne, and R. Watson, The Architecture of the High Performance Storage System, Fourth NASA GSFC Conference on Mass Storage Systems and Technologies, College Park, MD, March 28-30.1995.

19. R. Hyer, R. Ruef, and R. Watson, High Performance Direct Network Data Transfers at the National Storage Laboratory, Proc. Twelfth IEFiE Symposium on Mass Storage Systems, Monterey, CA, April 2629,1993.

20. S. Louis, and D. Teaff, Class of Service in the High Performance Storage System, Thiid TC6 International Conference on Open DistributedIFIP Processing, Brisbane, Australia, February 21-24, 1995.

21. J. Weinstock, Management of Distributed Aapiications: A Deveioper's Perspective, Conference Proceedings of the Enterprise Management Summit, San Jose, CA, November 14-18.1994. 22. D. Fisher and T. Tyler, Using Distributed Technology in the High Performance StorageOLTP System, Proc. of the 14th IEEE Computer Society Mass Storage Systems Symposium, Monterey, CA, September 11-14.1995. 23. S. Louis and S. Hyer, Applying IEEE Storage System Management Standards at the National Storage Laboratory, Proc. Twelfth IEEE Symposium on Mass Storage Systems, Mmterey, CA, April 2629,1993.