DOCSLIB.ORG

DATARMOR and New Tools.Pdf

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
DATARMOR and New Tools.Pdf JOURNÉE DE RENCONTRE DES UTILISATEURS DU PÔLE DE CALCUL INTENSIF POUR LA MER PIERRE COTTY Le programme de la journée • 9 h 40 - 12 h 10 DATARMOR: nouveau périmètre, nouvelles perspectives • Pause déjeuner : buffet • 14 h 00 - 16 h 30 DATARMOR: nouveaux champs d'applications scientifiques • Poster et Discussions autour d’un pot 17/05/2018 2 DATARMOR AND N E W T O O L S Tina ODAKA DATARMOR 17/05/2018 4 ACCES CLIENTS (10 Gigabit Ethernet connected to IFREMER and INFUSER network (SHOM / IUEM / ENSTA Bretagne Cluster HPC 426 Tflops 11 088 cores 396 Nodes Login nodes PBS Pro; sharing computing Batch system: resources PBS Pro InfiniBand FDR 17/05/2018 5 DATARMOR: not only HPC Cluster SMP Cluster Cluster HPC VIS High speed Cluster SCRATCH Network WEB Infini Band DATA DATAREF HOME DATA 17/05/2018 WORK 6 DISK • DATAREF (LUSTRE, 1.5PB = 1395TiB) • Reference data (next presentation) • DATAWORK (GPFS, 5PB = 4500TiB) • Work data • User space and project space • SCRATCH (LUSTRE 0.5PB) PB ? TiB?? • Scratch file for HPCcomparisons computation. with International System of Units (SI) prefixes • 15 days max one kibibit 1 Kibit = 210 bit = 1024 bit one kilobit 1 kbit = 103 bit = 1000 bit • DATAHOME (NFS, 0.04PB)one mebibyte 1 MiB = (210)2 B = 1 048 576 B • Source codes and softwaresone megabyte 1 MB = (103)2 B = 1 000 000 B • with backup ( tape, inone a gibibytedifferent1 GiBbuilding= (210)3 B) = 1 073 741 824 B one gigabyte 1 GB = (103)3 B = 1 000 000 000 B byte = octet( French) TiB (tebibyte) = Tio (French tébioctet) 17/05/2018 7 Clustre WEB, SMP, VIS • Cluster SMP • 1 node with 240 cores, 5 TB RAM • PBS Pro • Jobs only for huge memory, or many CPU cores. • Cluster VIS • 2 nodes each with: • 2 GPU (NVIDIA Quadro K6000: 2880 cuda cores, 12GB memory) • 2 Intel Cpu 2.2 Ghz each with 10 cores • PBS Pro • Visualisation. • Cluster WEB • 10 servers • Based on vmware virtual machine, and soon in docker/kubernetes • Hosting web services for datarmor, like VRE 17/05/2018 8 New tools on DATARMOR 17/05/2018 9 VRE: virtual research enviroment Complex system but simple access SeaDatacloud, Emodnet … 17/05/2018 10 Traditional access ‘ssh’ : terminal, command line,,, Must have basic linux knowledge. 17/05/2018 11 Challenge DATARMOR more flexible (web, click, ,, ) access Less of : vi, ssh, fortran, mpi Flexible, and easy access should speed up the learning/development/analysis time for users. It can lower the barrier for ‘new users’ non traditional users. It might slowdown computing time, but if it globally wins the time from development through having ‘result’ in a graph to be putted in a paper, whats the harm of slowing down a bit of computing time? How to co-habite traditional HPC users and new users? 17/05/2018 12 https://datarmor-vis.Ifremer.fr first step for VRE: example Visit Remote access to GPU Co-working possibilities PBS Pro controls the access. Cohabitation with traditional usage 17/05/2018 13 https://datarmor-web.Ifremer.fr • Jupyter notebook: interactive usage of computing power through PBS Pro • Netcdf related graphical tools (python) • Julia https://julialang.org • high performance, altenative of python?? • DIVA (fortran-mpi to Julia) • Pyspark • session spark end of this morning! • Machine learning tools through conda • Or other GUI tools… Compmression of image: interface by 17/05/2018 ‘nomachine’ software. 14 Virtual environment: integration with PBSPRO GARAXY @ DATARMOR Benefits: Bioinformatics web-tools • Light weight • Can share environment between different architectures. • Keep same environments as your PC/datarmor Implementations @datamor • conda • IFREMER developed Docker integration to PBS pro. PBS Pro controls the access. Cohabitation with traditional usage DATARMOR: status of use 17/05/2018 16 Reserved resources en K hours/month 6000 5000 4000 3000 2000 1000 0 mai-17 juin-17 juil.-17 août-17 sept.-17 oct.-17 nov.-17 déc.-17 janv.-18 févr.-18 mars-18 avr.-18 17/05/2018 17 Reserved CPU cores resources (%) 80,00% 70,00% 60,00% 50,00% 40,00% 30,00% 20,00% 10,00% 0,00% juil.-17 août-17 sept.-17 oct.-17 nov.-17 déc.-17 janv.-18 févr.-18 mars-18 avr.-18 HPCSIMPLE HPC HYPERHREAD SMP VISU 17/05/2018 18 Use of DATAWORK (TiB) 4000 MAX 3500 4500 TiB 3000 2500 2000 1500 1000 500 0 juil.-17 août-17 sept.-17 oct.-17 nov.-17 déc.-17 janv.-18 févr.-18 mars-18 avr.-18 Datawork alloué (Tio) Datawork occupé (Tio) 17/05/2018 19 Upgrade 2018 • Increase the capacity of datawork 1.5PB =1364TiB • cluster-web • Summer 2018 17/05/2018 20 DATARMOR is not only HPC DATARMOR and (Virtual) research environment Upgrade 2018 coming SOON THANK YOU 17/05/2018 21.
Load more

Recommended publications
  • Etir Code Lists
    eTIR Code Lists Code lists CL01 Equipment size and type description code (UN/EDIFACT 8155) Code specifying the size and type of equipment. 1 Dime coated tank A tank coated with dime. 2 Epoxy coated tank A tank coated with epoxy. 6 Pressurized tank A tank capable of holding pressurized goods. 7 Refrigerated tank A tank capable of keeping goods refrigerated. 9 Stainless steel tank A tank made of stainless steel. 10 Nonworking reefer container 40 ft A 40 foot refrigerated container that is not actively controlling temperature of the product. 12 Europallet 80 x 120 cm. 13 Scandinavian pallet 100 x 120 cm. 14 Trailer Non self-propelled vehicle designed for the carriage of cargo so that it can be towed by a motor vehicle. 15 Nonworking reefer container 20 ft A 20 foot refrigerated container that is not actively controlling temperature of the product. 16 Exchangeable pallet Standard pallet exchangeable following international convention. 17 Semi-trailer Non self propelled vehicle without front wheels designed for the carriage of cargo and provided with a kingpin. 18 Tank container 20 feet A tank container with a length of 20 feet. 19 Tank container 30 feet A tank container with a length of 30 feet. 20 Tank container 40 feet A tank container with a length of 40 feet. 21 Container IC 20 feet A container owned by InterContainer, a European railway subsidiary, with a length of 20 feet. 22 Container IC 30 feet A container owned by InterContainer, a European railway subsidiary, with a length of 30 feet. 23 Container IC 40 feet A container owned by InterContainer, a European railway subsidiary, with a length of 40 feet.
    [Show full text]
  • Course Conventions Fall 2016
    CS168 Computer Networks Fonseca Course Conventions Fall 2016 Contents 1 Introduction 1 2 RFC Terms 1 3 Data Sizes 2 1 Introduction This document covers conventions that will be used throughout the course. 2 RFC Terms For the project specifications in this class, we’ll be using proper RFC terminology. It’s the terminology you’ll see if you ever implement protocols in the real world (e.g., IMAP or MCTCP), so it’s good to get exposed to it now. In particular, we’ll be using the keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” as defined in RFC 2119. The terms we’ll use the most in this class are “MUST”, “MUST NOT”, “SHOULD”, “SHOULD NOT”, and “MAY” (though we may use others occasionally), so we’re including their definitions here for convenience (copied verbatim from the RFC): • MUST This word, or the terms “REQUIRED” or “SHALL”, mean that the definition is an absolute requirement of the specification. • MUST NOT This word, or the phrase “SHALL NOT”, mean that the definition is an absolute prohibition of the specification. • SHOULD This word, or the adjective “RECOMMENDED”, mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. • SHOULD NOT This phrase, or the phrase “NOT RECOMMENDED”, mean that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label.
    [Show full text]
  • Bitmath Documentation Release 1.3.3
    bitmath Documentation Release 1.3.3 Tim Bielawa Sep 13, 2018 Contents 1 Installation 3 2 Contents 5 2.1 The bitmath Module..........................................5 2.2 The bitmath command-line Tool................................... 19 2.3 Classes.................................................. 20 2.4 Instances................................................. 23 2.5 Getting Started.............................................. 29 2.6 Real Life Examples........................................... 32 2.7 Contributing to bitmath......................................... 41 2.8 Appendices................................................ 45 2.9 NEWS.................................................. 51 2.10 Contact.................................................. 56 2.11 Copyright................................................. 57 2.12 Examples................................................. 57 Python Module Index 63 i ii bitmath Documentation, Release 1.3.3 bitmath simplifies many facets of interacting with file sizes in various units. Originally focusing on file size unit conversion, functionality now includes: • Converting between SI and NIST prefix units (kB to GiB) • Converting between units of the same type (SI to SI, or NIST to NIST) • Automatic human-readable prefix selection (like in hurry.filesize) • Basic arithmetic operations (subtracting 42KiB from 50GiB) • Rich comparison operations (1024 Bytes == 1KiB) • bitwise operations (<<, >>, &, |, ^) • Reading a device’s storage capacity (Linux/OS X support only) • argparse integration as
    [Show full text]
  • Prefixes for Binary Multiples
    Definitions of the SI units: The binary prefixes Page 1 of 3 Return to Prefixes for binary multiples Units home page In December 1998 the International Electrotechnical Commission (IEC), the leading international organization for worldwide Units standardization in electrotechnology, approved as an IEC Topics: International Standard names and symbols for prefixes for binary Introduction multiples for use in the fields of data processing and data Units transmission. The prefixes are as follows: Prefixes Outside Prefixes for binary multiples Rules Background Factor Name Symbol Origin Derivation 210 kibi Ki kilobinary: (210)1 kilo: (103)1 Units 220 mebi Mi megabinary: (210)2 mega: (103)2 Bibliography 230 gibi Gi gigabinary: (210)3 giga: (103)3 Constants, 240 tebi Ti terabinary: (210)4 tera: (103)4 Units & 50 pebi Pi 10 5 3 5 Uncertainty 2 petabinary: (2 ) peta: (10 ) home page 260 exbi Ei exabinary: (210)6 exa: (103)6 Examples and comparisons with SI prefixes one kibibit 1 Kibit = 210 bit = 1024 bit one kilobit 1 kbit = 103 bit = 1000 bit one mebibyte 1 MiB = 220 B = 1 048 576 B one megabyte 1 MB = 106 B = 1 000 000 B one gibibyte 1 GiB = 230 B = 1 073 741 824 B one gigabyte 1 GB = 109 B = 1 000 000 000 B It is suggested that in English, the first syllable of the name of the binary-multiple prefix should be pronounced in the same way as the first syllable of the name of the corresponding SI prefix, and that the second syllable should be pronounced as "bee." It is important to recognize that the new prefixes for binary multiples are not part of the International System of Units (SI), the modern metric system.
    [Show full text]
  • The 2016 SNIA Dictionary
    A glossary of storage networking data, and information management terminology SNIA acknowledges and thanks its Voting Member Companies: Cisco Cryptsoft DDN Dell EMC Evaluator Group Fujitsu Hitachi HP Huawei IBM Intel Lenovo Macrosan Micron Microsoft NetApp Oracle Pure Storage Qlogic Samsung Toshiba Voting members as of 5.23.16 Storage Networking Industry Association Your Connection Is Here Welcome to the Storage Networking Industry Association (SNIA). Our mission is to lead the storage industry worldwide in developing and promoting standards, technologies, and educational services to empower organizations in the management of information. Made up of member companies spanning the global storage market, the SNIA connects the IT industry with end-to-end storage and information management solutions. From vendors, to channel partners, to end users, SNIA members are dedicated to providing the industry with a high level of knowledge exchange and thought leadership. An important part of our work is to deliver vendor-neutral and technology-agnostic information to the storage and data management industry to drive the advancement of IT technologies, standards, and education programs for all IT professionals. For more information visit: www.snia.org The Storage Networking Industry Association 4360 ArrowsWest Drive Colorado Springs, Colorado 80907, U.S.A. +1 719-694-1380 The 2016 SNIA Dictionary A glossary of storage networking, data, and information management terminology by the Storage Networking Industry Association The SNIA Dictionary contains terms and definitions related to storage and other information technologies, and is the storage networking industry's most comprehensive attempt to date to arrive at a common body of terminology for the technologies it represents.
    [Show full text]
  • Units of Measure Used in International Trade Page 1/57 Annex II (Informative) Units of Measure: Code Elements Listed by Name
    Annex II (Informative) Units of Measure: Code elements listed by name The table column titled “Level/Category” identifies the normative or informative relevance of the unit: level 1 – normative = SI normative units, standard and commonly used multiples level 2 – normative equivalent = SI normative equivalent units (UK, US, etc.) and commonly used multiples level 3 – informative = Units of count and other units of measure (invariably with no comprehensive conversion factor to SI) The code elements for units of packaging are specified in UN/ECE Recommendation No. 21 (Codes for types of cargo, packages and packaging materials). See note at the end of this Annex). ST Name Level/ Representation symbol Conversion factor to SI Common Description Category Code D 15 °C calorie 2 cal₁₅ 4,185 5 J A1 + 8-part cloud cover 3.9 A59 A unit of count defining the number of eighth-parts as a measure of the celestial dome cloud coverage. | access line 3.5 AL A unit of count defining the number of telephone access lines. acre 2 acre 4 046,856 m² ACR + active unit 3.9 E25 A unit of count defining the number of active units within a substance. + activity 3.2 ACT A unit of count defining the number of activities (activity: a unit of work or action). X actual ton 3.1 26 | additional minute 3.5 AH A unit of time defining the number of minutes in addition to the referenced minutes. | air dry metric ton 3.1 MD A unit of count defining the number of metric tons of a product, disregarding the water content of the product.
    [Show full text]
  • What Is a Computer? Foundations of Computer Science E.H
    What is a Computer? Foundations of Computer Science E.H. Turner and R.M. Turner If you are reading this book, it is almost certain you are very familiar with computers. You probably own at least one (laptop, desktop, cell phone. ), and you likely use them most days, probably for quite a few hours per day. It is possible, however, to be familiar with something without really knowing how it works, or even what is truly is. So what is a computer? At one level, a computer is a tool. Like any tool, it allows us to do things better than we could without it, or even to do things we could not do at all prior to the tool’s invention. Originally, and still to a large extent, computers augment our ability to do calculations: whereas we may be able to do one or two mathematical operations per second, with a computer, we can do billions; by ourselves, we may be able to laboriously do complex calculations, but with a computer, it becomes easy; by ourselves, and with pencil and paper, we may be able to remember quite a few data points, but with a computer, we can remember a virtually unlimited amount. Computers do more than math, of course. In fact, they can do so many things, including controlling things in the real world, that computers have been called the “universal tool”. We are all familiar with the myriad things that this tool enables us to do: write papers, carry hundreds of books in our pocket and read them, talk over long distances (cell phones, the Internet), listen to music, create music, watch movies, create movies, play video games, construct things (3D printers, robots), drive effortlessly (modern cruise control, autonomous cars), explore other planets and the deep sea, defend ourselves or attack others (torpedoes, drones), and on and on.
    [Show full text]
  • Course Conventions Fall 2017
    CS168 Computer Networks Fonseca Course Conventions Fall 2017 Contents 1 Introduction 1 2 RFC Terms 1 3 Data Sizes 2 1 Introduction This document covers conventions that will be used throughout the course. 2 RFC Terms For the project specifications in this class, we’ll be using proper RFC terminology. It’s the terminology you’ll see if you ever implement protocols in the real world (e.g., IMAP or MCTCP), so it’s good to get exposed to it now. In particular, we’ll be using the keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” as defined in RFC 2119. The terms we’ll use the most in this class are “MUST”, “MUST NOT”, “SHOULD”, “SHOULD NOT”, and “MAY” (though we may use others occasionally), so we’re including their definitions here for convenience (copied verbatim from the RFC): • MUST This word, or the terms “REQUIRED” or “SHALL”, mean that the definition is an absolute requirement of the specification. • MUST NOT This word, or the phrase “SHALL NOT”, mean that the definition is an absolute prohibition of the specification. • SHOULD This word, or the adjective “RECOMMENDED”, mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. • SHOULD NOT This phrase, or the phrase “NOT RECOMMENDED”, mean that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label.
    [Show full text]
  • File Organization & Management
    1 UNESCO -NIGERIA TECHNICAL & VOCATIONAL EDUCATION REVITALISATION PROJECT -PHASE II NATIONAL DIPLOMA IN COMPUTER TECHNOLOGY File Organization and Management YEAR II- SE MESTER I THEORY Version 1: December 2008 1 2 Table of Contents WEEK 1 File Concepts .................................................................................................................................6 Bit: . .................................................................................................................................................7 Binary digit .....................................................................................................................................8 Representation ...............................................................................................................................9 Transmission ..................................................................................................................................9 Storage ............................................................................................................................................9 Storage Unit .....................................................................................................................................9 Abbreviation and symbol ............................................................................................................ 10 More than one bit ......................................................................................................................... 11 Bit, trit,
    [Show full text]
  • The Complete List of Measurement Units Supported
    28.3.2019 The Complete List of Units to Convert The List of Units You Can Convert Pick one and click it to convert Weights and Measures Conversion The Complete List of Measurement Units Supported # A B C D E F G H I J K L M N O P Q R S T U V W X Y Z # ' (minute) '' (second) Common Units, Circular measure Common Units, Circular measure % (slope percent) % (percent) Slope (grade) units, Circular measure Percentages and Parts, Franctions and Percent ‰ (slope permille) ‰ (permille) Slope (grade) units, Circular measure Percentages and Parts, Franctions and Percent ℈ (scruple) ℔, ″ (pound) Apothecaries, Mass and weight Apothecaries, Mass and weight ℥ (ounce) 1 (unit, point) Apothecaries, Mass and weight Quantity Units, Franctions and Percent 1/10 (one tenth or .1) 1/16 (one sixteenth or .0625) Fractions, Franctions and Percent Fractions, Franctions and Percent 1/2 (half or .5) 1/3 (one third or .(3)) Fractions, Franctions and Percent Fractions, Franctions and Percent 1/32 (one thirty-second or .03125) 1/4 (quart, one forth or .25) Fractions, Franctions and Percent Fractions, Franctions and Percent 1/5 (tithe, one fifth or .2) 1/6 (one sixth or .1(6)) Fractions, Franctions and Percent Fractions, Franctions and Percent 1/7 (one seventh or .142857) 1/8 (one eights or .125) Fractions, Franctions and Percent Fractions, Franctions and Percent 1/9 (one ninth or .(1)) ångström Fractions, Franctions and Percent Metric, Distance and Length °C (degrees Celsius) °C (degrees Celsius) Temperature increment conversion, Temperature increment Temperature scale
    [Show full text]
  • Byte, Kilobyte Quick Reference Chart
    Byte, Kilobyte Quick Reference Chart TERM STANDARD ABBREVIATION DESCRIPTION VALUE bit JEDEC & IEC b Represents a single unit of digital information that can be one of two values: 0 or 1. The 1 b term “bit” is shorthand for binary digit. (can be 0 or 1) byte JEDEC & IEC B Generally consists of eight (8) bits of digital information as a block. 8 b kilobyte1 JEDEC KB Consists of 1024 bytes of data as one block. 1024 B kilobyte2 METRIC kB Consists of 1000 bytes of data as one block. 1000 B kibibyte IEC KiB Consists of 1024 bytes of data as one block. 1024 B megabyte1 JEDEC MB Consists of 1024 kilobytes of data as one block. 1024 KB megabyte2 METRIC MB Consists of 1000 kilobytes of data as one block. 1000 kB mebibyte IEC MiB Consists of 1024 kibibytes of data as one block. 1024 KiB gigabyte1 JEDEC GB Consists of 1024 megabytes of data as one block. 1024 MB gigabyte2 METRIC GB Consists of 1000 megabytes of data as one block. 1000 MB gibibyte IEC GiB Consists of 1024 mebibytes of data as one block. 1024 MiB terabyte1 JEDEC TB Consists of 1024 gigabytes of data as one block. 1024 GB terabyte2 METRIC TB Consists of 1000 gigabytes of data as one block. 1000 GB tebibyte IEC TiB Consists of 1024 gibibytes of data as one block. 1024 GiB petabyte1 JEDEC PB Consists of 1024 terabytes of data as one block. 1024 TB petabyte2 METRIC PB Consists of 1000 terabytes of data as one block. 1000 TB pebibyte IEC PiB Consists of 1024 tebibytes of data as one block.
    [Show full text]
  • AQA Computer Science A-Level 4.5.3 Units of Information Advanced Notes
    AQA Computer Science A-Level 4.5.3 Units of information Advanced Notes www.pmt.education Specification: 4.5.3.1 Bits and bytes: Know that: ● the bit is the fundamental unit of information ● a byte is a group of 8 bits n Know that the 2 ​ different values can be represented with n bits. ​ 4.5.3.2 Units: Know that quantities of bytes can be described using binary prefixes representing powers of 2 or using decimal prefixes representing powers of 10 10, eg one kibibyte is written as 1KiB = 2 ​ B and one kilobyte is written as ​ 3 1kB = 10 ​ B. ​ Know the names, symbols and corresponding powers of 2 for the binary prefixes: ● kibi, Ki - 210 ​ ● mebi, Mi - 220 ​ ● gibi, Gi - 230 ​ ● tebi, Ti - 240 ​ Know the names, symbols and corresponding powers of 10 for the decimal prefixes: ● kilo, k - 103 ​ ● mega, M - 106 ​ ● giga, G - 109 ​ ● tera, T - 1012 ​ www.pmt.education Bits and bytes A bit is the fundamental unit of information that can only take two values, 1 and 0, which ​ ​ ​ ​ ​ ​ can be represented by computers using high or low currents. ​ ​ A collection of 8 bits is called a byte. Half a byte (4 bits) is called a nybble. ​ ​ ​ ​ ​ ​ A bit is notated with a lowercase b whereas a byte uses the uppercase. ​ ​ ​ ​ 2b = 2 bits 3B = 3 bytes = 3 * 8 bits = 24 bits The number of different values that can be represented with a specified number of bits ​ ​ ​ varies with the number of bits. The more bits that are assigned to a number, the greater the number of values that can be represented.
    [Show full text]