C28b Mass Storage: Reliability and Efficiency Hui Chen Department of Computer & Information Science CUNY Brooklyn College
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
The Title Title: Subtitle March 2007
sub title The Title Title: Subtitle March 2007 Copyright c 2006-2007 BSD Certification Group, Inc. Permission to use, copy, modify, and distribute this documentation for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. THE DOCUMENTATION IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS DOCUMENTATION INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CON- SEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEG- LIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS DOCUMENTATION. NetBSD and pkgsrc are registered trademarks of the NetBSD Foundation, Inc. FreeBSD is a registered trademark of the FreeBSD Foundation. Contents Introduction vii 1 Installing and Upgrading the OS and Software 1 1.1 Recognize the installation program used by each operating system . 2 1.2 Recognize which commands are available for upgrading the operating system 6 1.3 Understand the difference between a pre-compiled binary and compiling from source . 8 1.4 Understand when it is preferable to install a pre-compiled binary and how to doso ...................................... 9 1.5 Recognize the available methods for compiling a customized binary . 10 1.6 Determine what software is installed on a system . 11 1.7 Determine which software requires upgrading . 12 1.8 Upgrade installed software . 12 1.9 Determine which software have outstanding security advisories . -
Disk Array Data Organizations and RAID
Guest Lecture for 15-440 Disk Array Data Organizations and RAID October 2010, Greg Ganger © 1 Plan for today Why have multiple disks? Storage capacity, performance capacity, reliability Load distribution problem and approaches disk striping Fault tolerance replication parity-based protection “RAID” and the Disk Array Matrix Rebuild October 2010, Greg Ganger © 2 Why multi-disk systems? A single storage device may not provide enough storage capacity, performance capacity, reliability So, what is the simplest arrangement? October 2010, Greg Ganger © 3 Just a bunch of disks (JBOD) A0 B0 C0 D0 A1 B1 C1 D1 A2 B2 C2 D2 A3 B3 C3 D3 Yes, it’s a goofy name industry really does sell “JBOD enclosures” October 2010, Greg Ganger © 4 Disk Subsystem Load Balancing I/O requests are almost never evenly distributed Some data is requested more than other data Depends on the apps, usage, time, … October 2010, Greg Ganger © 5 Disk Subsystem Load Balancing I/O requests are almost never evenly distributed Some data is requested more than other data Depends on the apps, usage, time, … What is the right data-to-disk assignment policy? Common approach: Fixed data placement Your data is on disk X, period! For good reasons too: you bought it or you’re paying more … Fancy: Dynamic data placement If some of your files are accessed a lot, the admin (or even system) may separate the “hot” files across multiple disks In this scenario, entire files systems (or even files) are manually moved by the system admin to specific disks October 2010, Greg -
CS 61C: Great Ideas in Computer Architecture Dependability: Parity
CS 61C: Great Ideas in Computer Architecture Dependability: Parity, RAID, ECC Instructor: Alan Christopher 8/07/2014 Summer 2014 -- Lecture #27 1 Review of Last Lecture • MapReduce Data Level Parallelism – Framework to divide up data to be processed in parallel – Handles worker failure and laggard jobs automatically – Mapper outputs intermediate (key, value) pairs – Optional Combiner in-between for better load balancing – Reducer “combines” intermediate values with same key 8/07/2014 Summer 2014 -- Lecture #27 2 Agenda • Dependability • Administrivia • RAID • Error Correcting Codes 8/07/2014 Summer 2014 -- Lecture #27 3 Six Great Ideas in Computer Architecture 1. Layers of Representation/Interpretation 2. Technology Trends 3. Principle of Locality/Memory Hierarchy 4. Parallelism 5. Performance Measurement & Improvement 6. Dependability via Redundancy 8/07/2014 Summer 2014 -- Lecture #27 4 Great Idea #6: Dependability via Redundancy • Redundancy so that a failing piece doesn’t make the whole system fail 2 of 3 agree 1+1=2 1+1=2 1+1=2 1+1=1 FAIL! 8/07/2014 Summer 2014 -- Lecture #27 5 Great Idea #6: Dependability via Redundancy • Applies to everything from datacenters to memory – Redundant datacenters so that can lose 1 datacenter but Internet service stays online – Redundant routes so can lose nodes but Internet doesn’t fail – Redundant disks so that can lose 1 disk but not lose data (Redundant Arrays of Independent Disks/RAID) – Redundant memory bits of so that can lose 1 bit but no data (Error Correcting Code/ECC Memory) 8/07/2014 Summer -
Identify Storage Technologies and Understand RAID
LESSON 4.1_4.2 98-365 Windows Server Administration Fundamentals IdentifyIdentify StorageStorage TechnologiesTechnologies andand UnderstandUnderstand RAIDRAID LESSON 4.1_4.2 98-365 Windows Server Administration Fundamentals Lesson Overview In this lesson, you will learn: Local storage options Network storage options Redundant Array of Independent Disk (RAID) options LESSON 4.1_4.2 98-365 Windows Server Administration Fundamentals Anticipatory Set List three different RAID configurations. Which of these three bus types has the fastest transfer speed? o Parallel ATA (PATA) o Serial ATA (SATA) o USB 2.0 LESSON 4.1_4.2 98-365 Windows Server Administration Fundamentals Local Storage Options Local storage options can range from a simple single disk to a Redundant Array of Independent Disks (RAID). Local storage options can be broken down into bus types: o Serial Advanced Technology Attachment (SATA) o Integrated Drive Electronics (IDE, now called Parallel ATA or PATA) o Small Computer System Interface (SCSI) o Serial Attached SCSI (SAS) LESSON 4.1_4.2 98-365 Windows Server Administration Fundamentals Local Storage Options SATA drives have taken the place of the tradition PATA drives. SATA have several advantages over PATA: o Reduced cable bulk and cost o Faster and more efficient data transfer o Hot-swapping technology LESSON 4.1_4.2 98-365 Windows Server Administration Fundamentals Local Storage Options (continued) SAS drives have taken the place of the traditional SCSI and Ultra SCSI drives in server class machines. SAS have several -
• RAID, an Acronym for Redundant Array of Independent Disks Was Invented to Address Problems of Disk Reliability, Cost, and Performance
RAID • RAID, an acronym for Redundant Array of Independent Disks was invented to address problems of disk reliability, cost, and performance. • In RAID, data is stored across many disks, with extra disks added to the array to provide error correction (redundancy). • The inventors of RAID, David Patterson, Garth Gibson, and Randy Katz, provided a RAID taxonomy that has persisted for a quarter of a century, despite many efforts to redefine it. 1 RAID 0: Striped Disk Array • RAID Level 0 is also known as drive spanning – Data is written in blocks across the entire array . 2 RAID 0 • Recommended Uses: – Video/image production/edition – Any app requiring high bandwidth – Good for non-critical storage of data that needs to be accessed at high speed • Good performance on reads and writes • Simple design, easy to implement • No fault tolerance (no redundancy) • Not reliable 3 RAID 1: Mirroring • RAID Level 1, also known as disk mirroring , provides 100% redundancy, and good performance. – Two matched sets of disks contain the same data. 4 RAID 1 • Recommended Uses: – Accounting, payroll, financial – Any app requiring high reliability (mission critical storage) • For best performance, controller should be able to do concurrent reads/writes per mirrored pair • Very simple technology • Storage capacity cut in half • S/W solutions often do not allow “hot swap” • High disk overhead, high cost 5 RAID 2: Bit-level Hamming Code ECC Parity • A RAID Level 2 configuration consists of a set of data drives, and a set of Hamming code drives. – Hamming code drives provide error correction for the data drives. -
Architectures and Algorithms for On-Line Failure Recovery in Redundant Disk Arrays
Architectures and Algorithms for On-Line Failure Recovery in Redundant Disk Arrays Draft copy submitted to the Journal of Distributed and Parallel Databases. A revised copy is published in this journal, vol. 2 no. 3, July 1994.. Mark Holland Department of Electrical and Computer Engineering Carnegie Mellon University 5000 Forbes Ave. Pittsburgh, PA 15213-3890 (412) 268-5237 [email protected] Garth A. Gibson School of Computer Science Carnegie Mellon University 5000 Forbes Ave. Pittsburgh, PA 15213-3890 (412) 268-5890 [email protected] Daniel P. Siewiorek School of Computer Science Carnegie Mellon University 5000 Forbes Ave. Pittsburgh, PA 15213-3890 (412) 268-2570 [email protected] Architectures and Algorithms for On-Line Failure Recovery In Redundant Disk Arrays1 Abstract The performance of traditional RAID Level 5 arrays is, for many applications, unacceptably poor while one of its constituent disks is non-functional. This paper describes and evaluates mechanisms by which this disk array failure-recovery performance can be improved. The two key issues addressed are the data layout, the mapping by which data and parity blocks are assigned to physical disk blocks in an array, and the reconstruction algorithm, which is the technique used to recover data that is lost when a component disk fails. The data layout techniques this paper investigates are variations on the declustered parity organiza- tion, a derivative of RAID Level 5 that allows a system to trade some of its data capacity for improved failure-recovery performance. Parity declustering improves the failure-mode performance of an array significantly, and a parity-declustered architecture is preferable to an equivalent-size multiple-group RAID Level 5 organization in environments where failure-recovery performance is important. -
PIC Serial Communication Modules
PIC Serial Communication Modules Synchronous Serial Port (SSP Module) {clock signal is required} - Serial Peripheral Interface (SPI) - Inter Integrated Circuit (IIC -> I2C) Serial Interface Asynchronous Communication Modules (clock signal is not required) - Universal Asynchronous Receiver Transmitter (UART) UART • A Universal Asynchronous Receiver-Transmitter (UART) is used for serial communications – usually via a cable. • The UART generates signals with the same timing as the RS-232 standard used by the Personal Computer’s COM ports. • The UART input/output uses 0V for logic 0 and 5V for logic 1. • The RS-232 standard (and the COM port) use +12V for logic 0 and –12V for logic 1. • To convert between these voltages levels we need an additional integrated circuit (such as Maxim’s MAX232). Parts of an RS-232 Frame • A frame transmits a single character and is in general composed of: 1) A start bit (always logic 0) 2) Data bits (5, 6, 7, or 8 of them) 3) A parity bit (optional, even or odd parity) 4) A stop bit (always logic 1) RS232 Level Converter PIC & MAX232 Connection UART Timing Accuracy • Since the transmitter and receiver keep track of time independently between clock recovery synchronization points, the combined inaccuracy of the transmitter and receiver’s clocks can not be too large. • For RS-232 communication, this combined inaccuracy is about 5%. • Usually, an RC oscillator is too inaccurate and a crystal is required. UART Bit Time and Baud Rate • The bit time (units of time) is the time from the start of one serial data bit value to the start of another. -
What the Heck Is RS-232 Anyway…
Tech Note What the heck is RS-232 anyway… It just occurred to me that there is an entire generation of technicians in the work force that did not grow up in the days of the TRS-80 and Commodore 64 PC. Rather, they were brought up on broadband and Wi-Fi. To them that funny looking 9-Pin connector that we old folks cut our teeth on is a mystery full of uncertainty and doubt – with a little bit of fear mixed in for good measure. In this paper, I hope to dispel some of the mystery and give you the essentials of RS-232. What is it? At one time, RS-232 / EIA-232 was the most widely used communication standard on the planet. It was defined and redefined many times. The “EIA” stands for “Electronic Industries Association” and the “RS” stands for “Recommended Standard.” That being the case, it was always rather loose. The physical characteristics of the hardware include both a 25 pin and 9 pin D sub connector. RS- 232 is capable of operating at data rates up to 20 Kbps and can push data about 50 ft. The absolute maximum data rate is difficult to nail down due the differences in the transmission line and cable length. It is possible to operate at some pretty high data rates if the distance is short. The voltage levels are defined as a range from -12 to +12 volts. RS-232 is also single ended. This means that a single electrical signal is compared to a common signal (ground) to determine binary logic states. -
Design and Its Application of Microprocessor
Design and Its application of Microprocessor The 8088 And 8086 Microprocessors: Programming, Interfacing, Software, Hardware And Applications Suk-Ju Kang Dong-A University [email protected] 1 Chapter 9. Memory Devices, Circuits, and Subsystem Design 2 In This Chapter, … 9.1 Program and Data Storage 9.2 Read-Only Memory 9.3 Random Access Read/Write Memories 9.4 Parity, the Parity Bit, and Parity- Checker/Generator Circuit 9.5 FLASH Memory 9.6 Wait-State Circuitry 9.7 8088/8086 Microcomputer System Memory Circuitry 3 Program and Data Storage The memory unit of a microcomputer is partitioned into a primary storage section and secondary storage section 4 Program and Data Storage The basic input/output system (BIOS) are programs held in ROM. ◦ They are called firmware because of their permanent nature ◦ The typical size of a BIOS ROM used in a PC today is 256 Kbytes Programs are normally read in from the secondary memory storage device, stored in the program storage part of memory, and then run 5 Read-Only Memory ROM, PROM, and EPROM ◦ Mask-programmable read-only memory (ROM) ◦ One-time-programmable read-only memory (PROM) ◦ Erasable read-only memory (EPROM) 6 Read-Only Memory Block diagram of a read-only memory Address bus ◦ Data bus ◦ Control bus Chip enable (CE) Output enable (OE) 7 Read-Only Memory Read operation 8 Random Access Read/Write Memories The memory section of a microcomputer system is normally formed from both read-only memories and random access read/write memories (RAM) RAM is different from ROM in two ways: ◦ Data stored in RAM is not permanent in nature RAM is normally used to store temporary data and application programs for execution ◦ RAM is volatile If power is removed from RAM, the stored data are lost 9 Random Access Read/Write Memories Static and dynamic RAMs ◦ For a static RAM (SRAM), data remain valid as long as the power supply is not turned off. -
Techsmart Representatives
Wave TechSmart representatives RAID BASICS ARE YOUR SECURITY SOLUTIONS FAULT TOLERANT? Redundant Array of Independent Disks (RAID) is a Enclosure: The "box" which contains the controller, storage technology used to improve the processing drives/drive trays and bays, power supplies, and fans is capability of storage systems. This technology is called an "enclosure." The enclosure includes various designed to provide reliability in disk array systems and controls, ports, and other features used to connect the to take advantage of the performance gains offered by RAID to a host for example. an array of mulple disks over single-disk storage. Wave RepresentaCves has experience with both high- RAID’s two primary underlying concepts are (1) that performance compuCng and enterprise storage, providing distribuCng data over mulple hard drives improves soluCons to large financial instuCons to research performance and (2) that using mulple drives properly laboratories. The security industry adopted superior allows for any one drive to fail without loss of data and compuCng and storage technologies aGer the transiCon without system downCme. In the event of a disk from analog systems to IP based networks. This failure, disk access will conCnue normally and the failure evoluCon has created robust and resilient systems that will be transparent to the host system. can handle high bandwidth from video surveillance soluCons to availability for access control and emergency Originally designed and implemented for SCSI drives, communicaCons. RAID principles have been applied to SATA and SAS drives in many video systems. Redundancy of any system, especially of components that have a lower tolerance in MTBF makes sense. -
RAID — Begin with the Basics How Does RAID Work? RAID Increases Data Protection and Performance by What Is RAID? Duplicating And/Or Spreading Data Over Multiple Disks
RAID — Begin With The Basics How Does RAID Work? RAID increases data protection and performance by What is RAID? duplicating and/or spreading data over multiple disks. DRIVE 1 RAID stands for Redundant Array of Inexpensive Disks. RAID is a method of logically treating Mirroring Mirroring & Striping several hard drives as one unit. It can offer fault tolerance and higher throughput levels than a Duplicates data from primary Mirrors data that is striped, spread single hard drive or group of independent hard drives. DRIVE 2 drive to secondary drive evenly across multiple disks Why Do We Need It? RAID provides real-time data recovery when a hard drive fails, increasing system uptime and DRIVE 1 DRIVE 1 DRIVE 3 availability while protecting against loss of data. Multiple drives working together also increase system performance. DRIVE 2 DRIVE 2 DRIVE 4 Levels of RAID DRIVE 1 DRIVE 3 RAID Level Description Minimum # of Drives Benefit RAID 0 Data striping (no data protection) 2 Highest performance DRIVE 2 DRIVE 4 RAID 1 Disk mirroring 2 Highest data protection RAID 1E Disk mirroring 3 Highest data protection for an odd number of disks RAID 5 Data striping with distributed parity 3 Best cost/performance balance for multi-drive environments RAID 5EE Data striping with distributed parity with 4 The cost/performance balance of RAID 5 without setting aside a dedicated hotspare disk hotspare integrated into the array RAID 6 Data striping with dual distributed parity 4 Highest fault tolerance with the ability to survive two disk failures RAID 10 Data -
Chapter 1 : " Introduction "
Chapter 1 : " Introduction " This chapter will introduce you to your new Disk Array's features and provide information on general RAID concept. 1-1 Introduction Features This section provides an overview of the features. For more detailed information, please refer to the technical specifications appendix at the end of this manual . Your Disk Array includes the following features : Easy Operation As everyone knows, conventional Disk Arrays are designed for experienced computer specialists. To solve complicated and time consuming operating procedures, we came up with a revolutionary idea : -- Innovative Plug And Play RAID -- As compared to a conventional Disk Array's long-winded setup procedures, your Disk Array can be ready to go after using the simple step by step built-in setup program. Ultra High performance Your Disk Array combines an extremely high speed microprocessor with the latest chip set, SCSI hardware technology, perfect firmware and an artistic design. The result is one of the fastest, most reliable Disk Array systems on the market. Supports virtually all popular operating systems ,platforms and network environments because it works independently from the O.S. Ultra 160 LVD SCSI channel interface to your Host computer, up to 160MB data transfer rate provides the processing and access power for you to handle complex and large files. Selective SCSI ID 0 ~ 14 , support with active termination. Tagged-command queuing : allows processing of up to 255 simultaneous data requests. Selective RAID levels 0, 1, 0+1, 3 or 5. Build-in 128MB cache memory, expandable up to 512MB. Serial communication port ( Terminal Port ) permits array controller operation through a standard VT100 terminal (or equivalent).