Three Data Delivery Cases for EMBL- EBI's Embassy

Home , CRAM (file format), InterPro, Protein Data Bank

Three data delivery cases for EMBL- EBI’s Embassy

Guy Cochrane www.ebi.ac.uk EMBL European Bioinformatics Institute

Genes, genomes & variation Protein sequences • European Nucleotide Archive • InterPro • 1000 Genomes • Pfam • Ensembl • UniProt • Ensembl Genomes • Ensembl Plants Molecular structures • European Genome-phenome Archive • Protein Data Bank in Europe • Metagenomics portal • Electron Microscopy Data Bank • GWAS Catalog browser

Expression • ArrayExpress Chemical biology • Expression Atlas • ChEMBL • Metabolights • ChEBI • PRIDE Literature & ontology • Europe PubMed Central Reactions, interactions • Gene Ontology & pathways Systems • Experimental Factor • IntAct • BioModels Ontology • Reactome • Enzyme Portal • MetaboLights • BioSamples Sequence data at EMBL-EBI

Sample/method Sample/method

Read Read

Alignment Alignment

European Genome-phenome Archive - Controlled access data - Human data around molecular medicine Assembly - http://www.ebi.ac.uk/ega/

Annotation

European Nucleotide Archive - Unrestricted data - Pan-species and application - http://www.ebi.ac.uk/ena/ Sequence data at EMBL-EBI

Sample/method Sample/method

Read Read

Alignment Alignment

European Genome-phenome Archive - Controlled access data - Human data around molecular medicine Assembly - http://www.ebi.ac.uk/ega/

Infrastructure provision Annotation - BBSRC: RNAcentral, MG Portal - MRC: 100k Genomes data implementation European Nucleotide Archive - EC: COMPARE, MicroB3, ESGI, - Unrestricted data BASIS - Pan-species and application - http://www.ebi.ac.uk/ena/ - etc. Challenges

• Data have high volume and grow rapidly • Data are dynamic (continuous feed) and their application has urgency • Users require arbitrary and ad hoc access Tara Oceans Tara Oceans

Capacity Infectious disease

• Opportunity: A methodological revolution in clinical and public health towards shotgun sequencing-based methods • Scientific power: Sequence harbours rich information • Diagnostic: identification, typing, resistance profiling, etc. • Public health: outbreak detection, response strategy, vaccine development

• Mechanistic: host interactions, pathogencity, virulence, transmission, anti- COMPARE: recently launched microbial resistance Horizon 2020 project in which EMBL-EBI is informatics provider

• Informatics roles for EMBL: • COMPARE: Rapid global sharing of surveillance and outbreak data, systematic integrated analysis, compute provision (Embassy) • Standards for reporting, analysis and the communication of results • New algorithms and analysis methods • User interfaces for surveillance data reporting , across the domains

Global Microbial Identifier: Initiative with EMBL-EBI involvement supporting technologies, standards and data sharing for pathogen surveillance COMPARE platform

Sources Processes Portals and environments

COMPARE COMPARE Data Resource COMPARE workﬂow engine Food COMPARE Portal Registry workﬂow development Public data Assembly & ‘Default’ tools alignment

Clinical INSDC data Annotaon ‘Hosted tools’ exchange workflow development API API Managed access Typing data Outbreak workflow development Private data Workflow integraon

API

EBI infrastructure Embassy infrastructure DTU infrastructure Embassy virtual domain COMPARE platform

Sources Processes Portals and environments

COMPARE COMPARE Data Resource COMPARE workﬂow engine Food COMPARE Portal Registry workﬂow development Public data Assembly & ‘Default’ tools alignment

Clinical INSDC data Annotaon ‘Hosted tools’ exchange workflow development API API Managed access Typing data Urgency Outbreak workflow development Private data Workflow integraon

API

EBI infrastructure Embassy infrastructure DTU infrastructure Embassy virtual domain Personalised medicine

• Motivation: Personalised studies of variation, cancer mutation, epigenetics, regulation, expression require references for comparison and interpretation

• As part of GA4GH, EMBL-EBI is working on • Resources serving reference human genomic and transcriptomic data, including Google read API, variant ‘Beacons’, etc. • CRAM compression supporting greater data ﬂuidity and APIs to allow direct computational access • Delivery and synchronisation of high volume datasets to local Embassy and remote cloud infrastructures • Past and current FP7 projects include SLING, BASIS, ESGI Personalised medicine

• Motivation: Personalised studies of variation, cancer mutation, epigenetics, regulation, expression require references for comparison and interpretation

• As part of GA4GH,Arbitrary EMBL-EBI is working access on • Resources serving reference human genomic and transcriptomic data, including Google read API, variant ‘Beacons’, etc. • CRAM compression supporting greater data ﬂuidity and APIs to allow direct computational access • Delivery and synchronisation of high volume datasets to local Embassy and remote cloud infrastructures • Past and current FP7 projects include SLING, BASIS, ESGI ENA conventional read data delivery

Conventional infrastructure (FTP, Aspera, GridFTP)

ENA metadata

FIRE1

ENA data (NFS) ENA Embassy read data delivery

Conventional infrastructure (FTP, Aspera, GridFTP)

ENA metadata

FIRE2 FUSE ENA data (Cleversafe)

HTTP ENA Embassy read data delivery

Conventional Embassy cloud infrastructure infrastructure (VMWare -> OpenStack) (FTP, Aspera, GridFTP)

Marine cache Tara Oceans Embassy

ENA metadata Pathogen cache COMPARE Embassy FIRE2 FUSE

ENA data CRAM (Cleversafe) cache GA4GH Embassy

HTTP ENA external read data delivery

…phase II EMBL-EBI Embassy Cloud

Steven Newhouse Head of Technical Services The Challenge Facing EMBL-EBI

• Volume and variety of genomic data expanding • EMBL-EBI data doubling every year - replication is challenging • Infrastructure currently 50,000 CPUs & 60+PB • Need to support complex analysis scenarios • Web and programmatic access to services (3M unique users) • Access to both public and managed access data sets • Bespoke workﬂows and tools across a variety of domains • Hard for users to replicate data sets for local analysis • Use the ‘cloud’ to bring local analysis to EMBL-EBI data

18 EMBL-EBI Embassy Cloud

• Service hosted at EMBL-EBI data centres • Direct network access to public and managed data sets • Direct network to access public services • Expect both academic and commercial users • Technical Implementation • Logically isolated outside EMBL-EBI’s LANs • Secure ﬂexible infrastructure for both tenant and host • Resources exposed using VMware’s vCloud Director & OpenStack • Provide isolated IaaS clouds to multiple users

19 Why ‘Embassy’ Cloud?

• An embassy is sovereign territory in a host country • Host Country: EMBL-EBI Data Centre • Sovereign Territory: Host Country not allowed to enter • Virtualisation provides the protection for ‘tenant’ and ‘host’ • Host puts boundaries in place to protect it from the tenant • Tenant has freedom and control within those boundaries

20 Embassy Cloud Concept PanCancer Public Data PublicData Private Data PrivateData Managed Data ManagedData Public Services PublicServices Embassy Cloud 1 EmbassyCloud 1 Embassy Cloud 3 EmbassyCloud 3 Embassy Cloud 2 EmbassyCloud 2

Virtualised EMBL-EBI Hardware

21 User Beneﬁts for the IaaS Model

• Tenant organisations get an empty virtual infrastructure • They establish their own virtual machines and networks • System administration performed by the tenant • EMBL-EBI staff have no access to the VMs • Added value from EMBL-EBI over other clouds • Machines and data hosted in known jurisdiction • Direct network data sets (public & managed access) • Direct network access to public EMBL-EBI services

22 Beneﬁts to EMBL-EBI of the IaaS Model

• A secure collaborative workspace • Work does not contend with main EMBL-EBI resources • Clearly deﬁne the committed IT resources and data • Explore how to build more data focused analysis services • Move the analysis to where the big data is located • Learn from and inform other big data scientiﬁc communities

23 Embassy Cloud: Typical Uses

• Collaborative Environment • Neutral ground outside internal network • CTTV: Resources and VMs to host intranet, databases, … • Data Staging • Undertake submission from local machine (following data staging) rather from remote location • BRAEMBL: Remote submission unreliable due to ﬁle upload • Data Analysis • Large scale management and analysis of data • PanCancer: 1,000 cores, 2.5 TB RAM, 1.0 PB HDD

Issues

• Object Store Storage Infrastructure • Essential for scalable high-performance storage • Applications need to adapt to ﬂat model • Current caching strategy will have a limit • Sharing resources between sites/communities/clouds • Adopt a standards based model for federating resources • Solutions for uploading and distributing VMs (+containers?) • Replicating large data sets to ‘attract’ workloads to a cloud

25 Gaps à Activities à Solutions?

• Data Set Replication • Strategic pre-positioning of data into clouds • Leverage JANET/GEANT, GridFTP + Globus Transfers, … • Cloud federation for mobile computing • EGI has a federated cloud and VM distribution model • ELIXIR plans to build on existing infrastructure where possible • Wide-area ﬁle access needed for collaborative data analysis • High performance wide-area object-store • Need access control for human related data • Coordinated investment in infrastructure • Where is the UK coordination? What coordination is needed? • Integrating commercial resources where they add value • Integration with EU Infrastructure (ELIXIR)