DOCSLIB.ORG

Convergence of HPC, Big Data and Machine Learning: a Science and Engineering Perspective

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Convergence of HPC, Big Data and Machine Learning: a Science and Engineering Perspective Convergence of HPC, Big Data and Machine Learning: A Science and Engineering Perspective Professor Tony Hey Chief Data Scientist Rutherford Appleton Laboratory, Science and Technology Facilities Council (STFC) [email protected] The Fourth Paradigm: Data-Intensive Science Thousand years ago – Experimental Science • Description of natural phenomena Last few hundred years – Theoretical Science . 2 • Newton’s Laws, Maxwell’s Equations… 2 a 4 cG a 3 a2 Last few decades – Computational Science • Simulation of complex phenomena Today – Data-Intensive Science • Scientists overwhelmed with data sets from many different sources • Data captured by instruments • Data generated by simulations • Data generated by sensor networks }." II11.1.·,..... ,. ~ -1 - -; ; ! '" f •• ' ·- ·' - r ~ .- -~~- . - ~ .. eScience is the set of tools and technologies I ---- .. _- to support data federation and collaboration • For analysis and data mining I f'i'II- • For data visualization and exploration • For scholarly communication and dissemination With thanks to Jim Gray Particle Physics and Astronomy kaggle Search kaggle Q. Competitions Datasets Kernels Discussion learn •·· Em Higgs Boson Machine Learning Challenge Higgs I.Ii' challenge l'I!. Use the ATLAS experiment to identify the Higgs boson $13 000 1 7 85 teams 4 ,ea sago Overv ,ev Data Discussion Leaderboard Rules Overview Description Run:204153 Evaluat ion >,('(;-~\ \ ATLAS ' 'l,,.~' Event: 35369265 Prizes Ji. EXPERIMENT 2012-05-30 20:31 :28 UTC About The Sponsors Timeline - / Winners Imperial College Data Science London Institute DeepJet: Jet classification with the CMS experiment Markus Stoye Imperial College London, DSI "Big data science in astropartide physics", HAP workshop, Aachen, Germany, 20th Feb. 2018 Partic e and vertex based DNN: DeepJet --700 400 250 ➔ ➔ ➔ • • • • • • • • charg. part . • • FC global -- 700 inputs and 250.000 model parameters • Particle and ve rtex based DN N has facto r 10 less free parameters than a generic Dense DN N wou ld have • 1OOM jets used fo r training, overtraining is not an issue Impact of DNN arch'tecture :ZS- cu .c 0._ C.. ~ 10- 1 E 10-3 ............................................. -...........__................................... -....--- .....................................................__..__ 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 b-jet efficiency Blue: generic DNN (650 inputs) e C QA e i uts) Red: Physics inspired DNN (650 inputs) Physics object based DN N performs best kaggle Search kaggle Q Competitions Datasets Kernels Discussion Learn •·· E9 25,000 Prize Money ~ CERN · 656 teams · 22 days ago · -- Overview Data Kernels Discussion Leaderboard Rules Overview Description To explore what our universe is made of, Evaluation scientists at CERN are colliding protons, essential ly recreating mini big bangs, and Prizes meticulously observing these collisions with intricate silicon detectors. About The Sponsors While orchestrating the collisions and Timeline observations is already a massive scientific accomplishment, analyzing the enormous amounts of data produced from the experiments is becoming an overwhelming challenge. Machine Learning in Astronomy Machine learning and the Dark Energy Survey (DES) - Classification: galaxy type (0908.2033), star/galaxy (1306.5236), Supernovae Ia (1603.00882) - Photo-z (1507.00490) - Mass of the Local Group (1606.02694) - Search for Planet 9 in DES Slides thanks to Ofer Lahar Photometric Classification of Supernovae , Data :n , , i -,,. , "I SALT2 Parametric models Wavelets .... ~ , NB KNN SVM AN BOT Classified s e Lochner, McEwen, Peiris, Lahav, Winter 00 • ; o.o o., 0.4 () 0.4 l 0 F I po tlv r arXiv: 1603.00882 . PLAsTiCC Astronomical Classification $25,000 Prize Money Can you help make sense of t he Universe? ~ LSST Pro;ect 34 teams 3 months to go (2 months ro go until lli!!fger dead~ne) Overvi~ v D el s.c ,e f -e world's le;;d1 ng astr omers Evaluation g asp the esDe., propgties of the u i\·e.rse. Prizes nmeline we've rc5'\•ill be The Pho:o ':tric SST ;.s• ~nomic Ti ,e-Sertes Cl-ilsifc.ation C alle•ge (Pl.As iCCj asks K:;.gg !HS to helporeparetocla..:,si thedatafra th.s ew survey.Cc petitors will l:.3s,· a: onomc so rec tat va,yw1 _me into di' e<Er,t clas.ses.. scali ng fro a srra,1 trai i•g s:t ta a very arg;; tet :et a the type e SS will dr.sCDVer. Acknowledgements The a 1mal Science FoJnda on NSF) 15 ac indepenO" t ·ea:ral age.rcy cn,aied by Congress i 1g50 :a µ•omoce tt.E p•ogre:;;; cf s.ci-= ~ NSF .suDpons a.;.ic r:5earch rd __ pie to create knawl~ge tha· i!.!'.:/a·ms. the· u·e. Home Contact Us Site Map oti iVacanaes SKA Science Site SQUARE KILOMETRE ARRAY Exploring the Universe with the world's largest radio telescope Choose your local minisite , -, _ __ _ _ - ~-~- ~ ~-~~=-~~ ~~ :- '-~q ~ -~' .-- /~--~~-:~~~ -~ ----~- iii f ~ 1111 Location Design Technology Science Industry Outreach & Education News & Media Technical Publications Recruitment Contacts Home » SKA Project 11!1 Pn t IS p Latest News SKA Project 22nd Dea: er 2015 2015: a big year for ASKAP! The Square Kilometre Array (SKA) project is an international 21 December 2015 effort to build the world's largest radio telescope, with Outcomes Of The 19th SKA eventually over a square kilometre (one million square Board Meeting metres) of collecting area. The scale of the SKA represents a huge leap forward in both er I and research & development towards building and delivering a unique instrument, with the detailed design and preparation now well under way. As one of the largest scientific endeavours in 7th December 2015 history, the SKA will bring together a wealth of the world's Artist impress,on of the Square Kilometre Allay Australia Announces AUS$293.7 finest scientists, engineers and policy makers to bring the Million for the SKA project to fruition. Data Flow through the SKA ~50 PFLOPS 8.8 Tb/s ~5 Tb/s SKA1-MID 7.2 Tb/s ~2 Pb/s 100 PFLOPS SKA1-LOW 130 - 300 PB/yr Users Exploring the Universe with t h e world's largest radio telescope D Footer text Big Scientific Data from Large Experimental Facilities in the UK Rutherford Appleton Laboratory ISIS (Spallation Central Laser LHC Tier 1 computing Neutron Source) Facility JASMIN Super-Data-Cluster Diamond Light Source Cryo-SXT Data Segmentation of Cryo-Soft X-ray Tomography (Cryo-SXT) data NucleousNucleus Data ● B24: Cryo Transmission X-ray Microscopy beamline at DLS ● Data Collection: Tilt series from ±65° with 0.5° step size ● Reconstructed volumes up to 1000x1000x600 voxels ● Voxel resolution: ~40nm currently ● Total depth: up to 10μm Cytoplasm ● GOAL: Study structure and morphological changes of whole cells Neuronal-like mammalian cell line; single slice The University of Nottingham Challenges: . diamond ● Noisy data, missing wedge artifacts, missing B24 beamline Computer Vision boundaries Data Analysis Software Group Laboratory ● Tens to hundreds of organelles per dataset ● Tedious to manually annotate 3D Volume Data Segmentation ● Cell types can look different ● Few previous annotations available ● Automated techniques usually fail .,_.... ~~ [email protected] ( Tomographic Cell Analysis: Feature Extraction Workflow Features are extracted from voxels to represent their appearance: Nucleo ● Intensity-based filters (Gaussian Convolutions) us Data ● Textural filters (eigenvalues of Hessian and Structure Tensor) Preprocessing User Annotation + Machine Learning Data Representation Feature Extraction User’s Manual Segmentations Predictions Refinement User Classification Annotations Using few user annotations as an input: ● Machine learning classifier (Random Forest) trained to discriminate between Refinement Nucleus and Cytoplasm and predict the class of each SuperVoxel ● Markov Random Field then used to refine the predictions [email protected] The OCTOPUS Imaging Village • Part of UKRI STFC Central Laser STFC-funded Facility at Harwell Campus Single Particle Imaging • Cluster of microscopes and lasers >5 nm resolution and expert end-to-end multidisciplinary support • National imaging facility with peer- reviewed, funded access ➢ Look at example of Single Molecule Super-resolution Microscopy ➢ Use technique of Fluorescence Localisation Imaging with Photobleaching (FLImP) Cellular Imaging ➢ 400 nm → 10 nm > 200 nm resolution Automated acquisition with Machine Learning Co Find good FLlmP ROIS Autor Suitable fluorescent t spot properties. Determine &sep photobleachlg needs. OUtput Cell segmentation Deep learning & Acquire FUmP data ~ Locate boundaries expllclt analysis. Time series of TIRF Images. of relevant cells. Automatically determine acquisition parameters. Scan sample Deep leamlng. Brlghtfleld & TIRF. Key task - identify ROIs to acquire FLImP data • Segment cell type of interest from in NSCLC biopsy • Fibroblasts, epithelial, endothelial, red blood Single molecules in an • Segment ROI good for single molecule FLImP epithelial cell from EBUS • Well focussed, separated spots with smooth background • Deep learning (CNNs) proven in cell segmentation • We wish to apply this to our system to identify the ROIs ROI = Region Of Interest FLImP Vision for Personalized Cancer Care @ . sv samples process b10P d irnager in automate Microscope slides 1 Cell Acquire segmentation FLlmPdata Patient biopsy @ :::::=:::: :.:~....· ~:·~ ~" '-----"""" ,,,,, .. ~,;,~.:-: .. ~-~ Patient-specific Patient database Machine learning classifiers Research database predictions & decisions Feedback patient outcomes Fully automate from acquisition to output • Increase efficiency and impact
Load more

Recommended publications
  • The UK E-Science Core Programme and the Grid Tony Hey∗, Anne E
    Future Generation Computer Systems 18 (2002) 1017–1031 The UK e-Science Core Programme and the Grid Tony Hey∗, Anne E. Trefethen UK e-Science Core Programme EPSRC, Polaris House, North Star Avenue, Swindon SN2 1ET, UK Abstract This paper describes the £120M UK ‘e-Science’ (http://www.research-councils.ac.uk/and http://www.escience-grid.org.uk) initiative and begins by defining what is meant by the term e-Science. The majority of the £120M, some £75M, is funding large-scale e-Science pilot projects in many areas of science and engineering. The infrastructure needed to support such projects must permit routine sharing of distributed and heterogeneous computational and data resources as well as supporting effective collaboration between groups of scientists. Such an infrastructure is commonly referred to as the Grid. Apart from £10M towards a Teraflop computer, the remaining funds, some £35M, constitute the e-Science ‘Core Programme’. The goal of this Core Programme is to advance the development of robust and generic Grid middleware in collaboration with industry. The key elements of the Core Programme will be outlined including details of a UK e-Science Grid testbed. The pilot e-Science projects that have so far been announced are then briefly described. These projects span a range of disciplines from particle physics and astronomy to engineering and healthcare, and illustrate the breadth of the UK e-Science Programme. In addition to these major e-Science projects, the Core Programme is funding a series of short-term e-Science demonstrators across a number of disciplines as well as projects in network traffic engineering and some international collaborative activities.
    [Show full text]
  • The Fourth Paradigm
    ABOUT THE FOURTH PARADIGM This book presents the first broad look at the rapidly emerging field of data- THE FOUR intensive science, with the goal of influencing the worldwide scientific and com- puting research communities and inspiring the next generation of scientists. Increasingly, scientific breakthroughs will be powered by advanced computing capabilities that help researchers manipulate and explore massive datasets. The speed at which any given scientific discipline advances will depend on how well its researchers collaborate with one another, and with technologists, in areas of eScience such as databases, workflow management, visualization, and cloud- computing technologies. This collection of essays expands on the vision of pio- T neering computer scientist Jim Gray for a new, fourth paradigm of discovery based H PARADIGM on data-intensive science and offers insights into how it can be fully realized. “The impact of Jim Gray’s thinking is continuing to get people to think in a new way about how data and software are redefining what it means to do science.” —Bill GaTES “I often tell people working in eScience that they aren’t in this field because they are visionaries or super-intelligent—it’s because they care about science The and they are alive now. It is about technology changing the world, and science taking advantage of it, to do more and do better.” —RhyS FRANCIS, AUSTRALIAN eRESEARCH INFRASTRUCTURE COUNCIL F OURTH “One of the greatest challenges for 21st-century science is how we respond to this new era of data-intensive
    [Show full text]
  • Mr Chancellor
    ANTHONY JOHN GRENVILLE HEY DCL Mr Chancellor, “All the world's a stage … and one man in his time plays many parts”. Of no-one is this truer than of Professor Tony Hey. He does indeed stride the world stage, and has successively been at the forefront of particle physics, computing science, research programme direction, the popularization of ‘hard science’, and the stimulation of innovation in the world’s foremost software company. Notwithstanding his thirty-one years of service to the University of Southampton in a succession of ever more senior academic posts, Tony’s most recent appointment (as Corporate Vice President of Technical Computing at Microsoft) actually marks the sixth time he has set up camp in the United States: one gets the impression that only the QE2 might exceed his record for round-trips between Southampton and the USA! The roots of his long-term fascination with America were laid immediately after he completed his PhD at Oxford, when he took up a Harkness Fellowship at Caltech to work at the forefront of particle physics with two Nobel Prize Winners: Richard Feynman and Murray Gell-Mann. The spirit of openness, drive and adventure with which his American colleagues were imbued made a lasting impression on Tony. 1 After leaving the USA, Tony spent two years in Geneva as a Research Fellow at CERN, the European Organization for Nuclear Research. Later to become the birthplace of the worldwide web, in the early 1970s CERN was rapidly expanding its unique facilities for probing subatomic particles, and Tony worked in a team which developed the theoretical under-pinning for much of modern quantum physics.
    [Show full text]
  • First Report of ASCAC-OSTI Subcommittee
    First Report of ASCAC-OSTI Subcommittee Tony Hey Subcommittee Chair eScience Institute University of Washington [email protected] What Is OSTI? OSTI is a program within DOE’s Office of Science, with a Since 1947! corporate responsibility for ensuring access to DOE R&D results. Public access to unclassified, unlimited Restricted access to classified and sensitive Slide courtesy of Brian Hitson DOE STI Program • OSTI manages agency-wide program – STIP • Responsibilities defined in DOE O 241.1B • Broad network of STI managers across DOE complex • R&D results collected electronically from Labs and Grantees • STI Submissions by year FY12 FY13 FY14 20,205 28,793 41,867 Slide courtesy of Brian Hitson Context: OSTP Memorandum • Directive requiring the major Federal Funding agencies “to develop a plan to support increased public access to the results of research funded by the Federal Government.” • The memorandum defines digital data “as the digital recorded factual material commonly accepted in the scientific community as necessary to validate research findings including data sets used to support scholarly publications, but does not include laboratory notebooks, preliminary analyses, drafts of scientific papers, plans for future research, peer review reports, communications with colleagues, or physical objects, such as laboratory specimens.” 22 February 2013 DOE Public Access Plan, July 24, 2014 Public Access to Scientific Publications • OSTI is responsible for operating and maintaining DOE’s public access system and network • The submission of
    [Show full text]
  • Agenda All Presentations Will Take Place in the Main Meeting Room in the Training Centre
    The PharmaGRID Retreat Organized by The PRISM Forum June 30th-July 2nd 2003 The Node Conference Centre, Welwyn, UK Agenda All presentations will take place in the main meeting room in the training centre. Lunch will be served in the Dining Room, which is located in the main building. Dinner on the evening on June 30th will be held in the main dining room, and BBQ will take place on the evening of July 1st. Pre-dinner drinks will be served in the bar or on the terrace outside the dining room if the weather is pleasant. The Node makes tea and coffee available throughout the day in the Bar and in the meeting rooms. Please help yourself. Coffee will be available during the coffee breaks in the Sponsor Room located directly opposite the main event room. June 30th- Setting the Scene 15:30 Tea/Coffee/Registration 16:15 Introduction Chris Jones 16:45 GRID Computing Prof. Tony Hey Professor Tony Hey, Director UK e -Science Core Programme and Editor of recent book on GRID Computing. For more information see: http://www.ecs.soton.ac.uk/~ajgh http://www.research-councils.ac.uk/escience 17:30 Particle Physics GRIDS Prof. Hans Hoffmann Director for Technology Transfer and Scientific Computing in the CERN directorate, responsible for the LHC Computing Grid. For more information see: http://www.twgrid.org/event/isgc2003/cv/hoffmann.pdf 18:15 BIRN Prof. Mark Ellisman Professor Mark Ellisman, Department of Neurosciences, UCSD, and Principal Investigator, Biomedical Information Research Network (BIRN). For more information see: http://www.sdsc.edu/Partners/Fellows/bios/mellisman.html 19:00 Open Discussion “Where do we want to go?” Chris Jones & Howard Bilofsky 19:30 Aperitifs 20:00 Dinner July 1st- The State of the Art 08:30 Chairman's Remarks Rene Ziegler 08:45 EU Funding for the GRID and directions Dr.
    [Show full text]
  • Constraint-Based Pattern Mining from Classic Pattern Domains to More Sophisticated Ones
    Constraint-Based Pattern Mining From classic pattern domains to more sophisticated ones Marc Plantevit October 2016 Universit´eClaude Bernard Lyon 1 { LIRIS CNRS UMR5205 * Slides from the ´eEGClecture. About me. [email protected] or [email protected] Associate Professor Computer Science Dept. University Claude Bernard Lyon 1. Lab: LIRIS UMR 5205 Team: Data Mining & Machine Learning Research Interest: Fondations of constraint-based pattern mining, sequences, augmented graphs. Before: Ph.D from University Montpellier II (LIRMM), Post-Doc at Univ. Caen (GREYC). M. Plantevit Constraint-Based Pattern Mining 2 / 118 N Introduction Outline 1 Introduction 2 Frequent Itemset Mining Frequent Itemset Mining Condensed Representations 3 Constraint-based Pattern Mining Constraint properties Algorithmic principles Constraint-based pattern mining with preferences 4 Toward More Sophisticated Pattern Domains Sequence, graphs, dense subgraphs Attributed Graph Mining 5 Conclusion M. Plantevit Constraint-Based Pattern Mining 3 / 118 N Introduction Evolution of Sciences Before 1600: Empirical Science Babylonian mathematics: 4 basis operations done with tablets and the resolution of practical problems based on words describing all the steps. ) that worked and they manage to solve 3 degree equations. Ancient Egypt: No theorization of algorithms. We give only examples made empirically, certainly repeated by students and scribes. Empirical knowledge, transmitted as such, and not a rational mathematical science. Aristotle also produced many biological writings that were empirical in nature, focusing on biological causation and the diversity of life. He made countless observations of nature, especially the habits and attributes of plants and animals in the world around him, classified more than 540 animal species, and dissected at least 50.
    [Show full text]
  • Jim Gray on Escience: a Transformed Scientific Method Based on the Transcript of a Talk Given by Jim Gray to the NRC-CSTB1 in Mountain View, CA, on January 11, 20072
    Jim Gray on eScience: A Transformed Scientific Method Based on the transcript of a talk given by Jim Gray to the NRC-CSTB1 in Mountain View, CA, on January 11, 20072 EDITED by TONY HEY, STEWART TANSLEY, AND KRIstIN TOLLE | Microsoft Research E HAVE TO DO BETTER AT PRODUCING TOOLS to support the whole re- search cycle—from data capture and data curation to data analysis and data visualization. Today, the tools for capturing data both at the mega-scale and at the milli-scale are just dreadful. After you Whave captured the data, you need to curate it before you can start doing any kind of data analysis, and we lack good tools for both data curation and data analysis. Then comes the publication of the results of your research, and the published literature is just the tip of the data iceberg. By this I mean that people collect a lot of data and then reduce this down to some number of column inches in Science or Nature—or 10 pages if it is a computer science person writing. So what I mean by data iceberg is that there is a lot of data that is collected but not curated or published in any systematic way. There are some exceptions, and I think that these cases are a good place for us to look for best practices. I will talk about how the whole process of peer review has got to change and the way in which I think it is changing and what CSTB can do to help all of us get access to our research.
    [Show full text]
  • Preference-Based Pattern Mining
    Preference-Based Pattern Mining Marc Plantevit Université Claude Bernard Lyon 1 – LIRIS CNRS UMR5205 * Slides from different research school lectures and conference tutorials. eBISS 2018 . About me. [email protected] or [email protected] Associate Professor, HDR Computer Science Dept. University Claude Bernard Lyon 1. Lab: LIRIS UMR 5205 Team: Data Mining & Machine Learning (head since 2019) Research Interest: Foundations of constraint-based pattern mining, sequences, augmented graphs, subgroup discovery, XAI. Evolution of Sciences Before 1600: Empirical Science I Babylonian mathematics: 4 basis operations done with tablets and the resolution of practical problems based on words describing all the steps. ⇒ that worked and they manage to solve 3 degree equations. I Ancient Egypt: No theorization of algorithms. We give only examples made empirically, certainly repeated by students and scribes. Empirical knowledge, transmitted as such, and not a rational mathematical science. I Aristotle also produced many biological writings that were empirical in nature, focusing on biological causation and the diversity of life. He made countless observations of nature, especially the habits and attributes of plants and animals in the world around him, classified more than 540 animal species, and dissected at least 50. I ... Wikipedia . 1600-1950s: Theoretical Science Each discipline has grown a theoretical component. Theoretical models often motivate experiments and generalize our understanding. I Physics: Newton, Max Planck, Albert Einstein, Niels Bohr, Schrödinger I Mathematics: Blaise Pascal, Newton, Leibniz, Laplace, Cauchy, Galois, Gauss, Riemann I Chemistry: R. Boyle, Lavoisier, Dalton, Mendeleev, I Biology, Medecine, Genetics: Darwin, Mendel, Pasteur . 1950s–1990s, Computational Science I Over the last 50 years, most disciplines have grown a third, computational branch (e.g.
    [Show full text]
  • E-Science, Virtual Organisations and the Grid
    e-Science, Virtual Organisations and the Grid Tony Hey, Director of UK e-Science Core Programme EPSRC, UK e-Science and the Grid ‘e-Science is about global collaboration in key areas of science, and the next generation of infrastructure that will enable it.’ John Taylor Director General of Research Councils Office of Science and Technology What is the Grid? • "A new technology infrastructure, which we call the GRID, will emerge to supersede the Web, and form the bedrock of the new economy" (Deutsche Bank) • "Web and computing services shall go the way of the electric power grid – always available, easily accessible, and predictably expandable" (PricewaterhouseCoopers) The Grid as an Enabler for Virtual Organisations !Ian Foster, Carl Kesselman and Steve Tueke ‘The Grid is a software infrastructure that enables flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions and resources’ !Enabling infrastructure for transient ‘Virtual Organisations’ not only for e-Science but also c-Commerce and e-Utilities NASA’s IPG Vision for the Information Power Grid is to promote a revolution in how NASA addresses large-scale science and engineering problems by providing persistent infrastructure for: – “highly capable” computing and data management services that, on-demand, will locate and co- schedule the multi-Center resources needed to address large-scale and/or widely distributed problems – the ancillary services that are needed to support the workflow management frameworks that coordinate the processes of distributed science and engineering problems Multi-disciplinary Simulations Multiple sub-systems, e.g. a wing lift model operating at NASA Ames and a turbo-machine model operating at NASA Glenn, are combined using an application framework that manages the interactions of multiple models and uses IPG services to coordinate computing and data storage systems across NASA.
    [Show full text]
  • The Fourth Paradigm 10 Years on }
    AKTUELLES SCHLAGWORT / THE FOURTH PARADIGM 10 YEARS ON } The Fourth Paradigm 10 Years On Tony Hey · Anne Trefethen Introduction each section below, together with a brief commen- The book, The Fourth Paradigm: Data-Intensive Sci- tary on how these projections compare with the entific Discovery, was a collection of provocative, situation in scientific research 10 years on. forward-looking essays published in 2009. It now seems a good time to look back at some of the sig- Earth and environment nificant developments in data-intensive science and – From Jeff Dozier and Bill Gail on “The Emerging scholarly publishing that have happened in the last Science of Environmental Applications” 10 years and see how the predictions of the authors The emerging third phase, knowledge developed have fared. primarily for the purpose of scientific understanding The book was dedicated to Turing Award winner is being complemented by knowledge created to Jim Gray of Microsoft Research, who was tragically target practical decisions and action. This new lost at sea in January 2007. Jim’s vision for this Fourth knowledge endeavor can be referred to as the science Paradigm for research had its origins nearly 10 years of environmental applications. earlier with Jim in the USA, and with “eScience” in – From Jim Hunt, Dennis Baldocchi and Catharine the UK. In the late 1990s Jim had recognized that the van Ingen on “Redefining Ecological Science Using next “Big Data” challenge for database technologies Data” would likely come from science rather than from These changes require a new approach to resolving commerce. He understood the very real technical resource management questions.
    [Show full text]
  • Toward Timely, Predictable and Cost-Effective Data Analytics
    Toward timely, predictable and cost-effective data analytics THÈSE NO 7140 (2016) PRÉSENTÉE LE 3 NOVEMBRE 2016 À LA FACULTÉ INFORMATIQUE ET COMMUNICATIONS LABORATOIRE DE SYSTÈMES ET APPLICATIONS DE TRAITEMENT DE DONNÉES MASSIVES PROGRAMME DOCTORAL EN INFORMATIQUE ET COMMUNICATIONS ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE POUR L'OBTENTION DU GRADE DE DOCTEUR ÈS SCIENCES PAR Renata BOROVICA-GAJIC acceptée sur proposition du jury: Prof. C. Koch, président du jury Prof. A. Ailamaki, directrice de thèse Dr S. Chaudhuri, rapporteur Dr G. Graefe, rapporteur Prof. K. Aberer, rapporteur Suisse 2016 "Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop questioning." —Albert Einstein To Ivan, my husband, my friend, my soul mate, for encouraging me to pursue my dreams, and never letting me give up. To Mia, our beautiful daughter, for lightening up my days and making me look once again at life as a miracle. To Tamara, my "big" sister and partner in crime, for being there whenever I needed her, and to my parents, Dijana and Petar, for letting me fly. I dedicate this thesis to them... Acknowledgements Although this thesis has only one name written under the tittle as the author, nothing could be further from the truth. This thesis would have not been possible without the help and support of my advisors, my collaborators, my friends, and my family, all of who helped me throughout this journey in their own ways. First and foremost, I would like to thank my advisor, Anastasia Ailamaki, for being an advisor in the real sense of the word.
    [Show full text]
  • Tony Hey Vice President Microsoft Research Big Data and the Fourth Paradigm Data Size and Speed Are Growing
    Tony Hey Vice President Microsoft Research Big Data and the Fourth Paradigm Data Size and Speed are Growing Large Hadron Collider Entire sequence of DNA for the 150 million sensors delivering Thousands of small antennas human body, consists of around data 40 million times per spread over a distance of more 3 billion of these base pairs. second. than 3000km. Data flow: ~700 MB/sec Data flow: ~60 GB/sec 1 Million PB/day The human genome requires ~15 PB/year ~750 megabytes of storage The SKA supercomputer will 1000’s of scientists around the perform 1018 operations per world; Institutions in 34 second ~ 100M PCs different countries: 1990 2000 2010 2020 Much of Science is now Data-Intensive Four “V’s” of Data • Extremely large data sets • Expensive to move • Volume • Domain standards • High computational needs • Supercomputers, HPC, Grids • Variety e.g. High Energy Physics, Astronomy • Velocity • • Large data sets Veracity Data Volume • Some Standards within Domains • Shared Datacenters & Clusters • Research Collaborations • Medium & Small data sets e.g. Genomics, Financial • Flat Files, Excel • Widely diverse data; Few standards • Local Servers & PCs e.g. Social Sciences, Humanities Number of Researchers ‘The Long Tail of Science’ Jim Gray, Turing Award Winner The ‘Cosmic Genome’ Project • The Sloan Digital Sky Survey was the first major astronomical survey project: • 5 color images and spectra of ¼ of the sky • Pictures of over 300 million celestial objects • Distances to the closest 1 million galaxies • Jim Gray from Microsoft Research worked with astronomer Alex Szalay to build the public ‘SkyServer’ archive for the survey New model of scientific publishing - publish the data before astronomers publish their analysis eScience and the Fourth Paradigm Thousand years ago – Experimental Science • Description of natural phenomena Last few hundred years – Theoretical Science .
    [Show full text]