Cloud Computing with Nimbus March 2009, OGF25
Total Page:16
File Type:pdf, Size:1020Kb
Cloud Computing with Nimbus March 2009, OGF25 Thilo Kielmann (slides by Kate Keahey [email protected]) Nimbus Nimbus goals Allow providers to build clouds Private clouds (privacy, expense considerations) E.g., Workspace Service: open source EC2 implementation Allow users to use cloud computing Do whatever it takes to enable scientists to use IaaS E.g.,Context Broker: creates turnkey virtual clusters Allow developers to experiment with Nimbus For research or usability/performance improvements Community extensions and contributions, e.g Ian Gable and his team (UVIC) contributed a monitoring component First released in September 2005 3/2/09 The Nimbus Toolkit: http//workspace.globus.org The Workspace Service Pool Pool Pool node node node VWS Service Pool Pool Pool node node node Pool Pool Pool node node node Pool Pool Pool node node node 3/2/09 The Nimbus Toolkit: http//workspace.globus.org The Workspace Service The workspace service publishes information about each workspace Pool Pool Pool node node node VWS Service Pool Pool Pool node node node Users can find out information about their Pool Pool Pool workspace (e.g. what IP node node node the workspace was bound to) Pool Pool Pool node node node Users can interact directly with their workspaces the same way the would with a physical machine. 3/2/09 The Nimbus Toolkit: http//workspace.globus.org Workspace Service: Interfaces and Clients Two kinds of interfaces: Web Service Resource Framework (WSRF) Web Services + state management (WS-Notification) Cloud client (similar commads to the EC2 client) Elastic Computing Cloud (EC2) Supported: ec2-describe-images, ec2-run-instances, ec2-describe- instances, ec2-terminate-instances, ec2-reboot-instances, ec2-add- keypair, ec2-delete-keypair Unsupported: availability zones, security groups, elastic IP assignment, REST Works with EC2 clients You can use one, or the other, or both 3/2/09 The Nimbus Toolkit: http//workspace.globus.org Workspace Service: Security Authenticating and authorizing requests WSRF interfaces: GSI Support for proxies, VOMS, Shibboleth (via GridShib), custom PDPs EC2 interfaces Vanilla PKI X509 support Secure access to VMs EC2 key generation or accessed from .ssh Towards protecting data from the provider Extensions from Vienna University of Technology: Paper: Descher et al., Retaining Data Control in Infrastructure Clouds, ARES (the International Dependability Conference), 2009. 3/2/09 The Nimbus Toolkit: http//workspace.globus.org Workspace Service: Networking Network configuration External: public IPs or private IPs (via VPN) Internal: private network via a local cluster network Each VM can specify multiple NICs mixing private and public networks (WSRF only) E.g., cluster worker nodes on a private network, headnode on both public and private network 3/2/09 The Nimbus Toolkit: http//workspace.globus.org Workspace Components workspace resource workspace manager WSRF workspace service control EC2 workspace pilot workspace OpenNebula client Project 3/2/09 The Nimbus Toolkit: http//workspace.globus.org Turnkey Virtual Clusters Turnkey, tightly-coupled cluster Shared trust/security context Shared configuration/context information Context Broker goals Every appliance Every cloud provider Multiple distributed cloud providers Used to contextualize 100s of virtual nodes for EC2 HEP STAR runs, Hadoop nodes, HEP Alice nodes… Working with rPath on developing appliances, standardization 3/2/09 The Nimbus Toolkit: http//workspace.globus.org Science Clouds Goals: Enable scientific projects to experiment with IaaS clouds Evolve software in response to the needs of scientific projects A laboratory for exploration of cloud interoperability issues Participants University of Chicago (since 03/08, 16 nodes), University of Florida (05/08, 16-32 nodes, access via VPN), Masaryk University, Brno, Czech Republic (08/08), Wispy @ Purdue (09/08) In progress: IU, Grid5K, Vrije, others Using EC2 for large runs http://workspace.globus.org/clouds 3/2/09 The Nimbus Toolkit: http//workspace.globus.org Who Runs on Nimbus? Hadoop AliEn GT-scalability STAR Montage workflows GridFTP testing workspace-team Testing OSG geofest bioinformatics Other Project diversity: Science, CS, education, build&test… 3/2/09 The Nimbus Toolkit: http//workspace.globus.org STAR STAR: a high-energy physics experiment Need resources with the right configuration Complex environments Consistent environments A virtual OSG STAR cluster OSG cluster: OSG CE (headnode), gridmapfiles, host certificates, NSF, Torque, worker nodes: SL4 + STAR Requirements One-click virtual cluster deployment Moving virtual clusters: Science Clouds -> EC2 From proof-of-concept to productions runs Work by Jerome Lauret, Doug Olson, Leve Hajdu, Lidia Didenko at BNL Signinficant productions runs in progress now: results to be published at Quark Matter conference and CHEP 3/2/09 The Nimbus Toolkit: http//workspace.globus.org Alice HEP Experiment at CERN Collaboration with CERNVM project 3/2/09 The Nimbus Toolkit: http//workspace.globus.org Making a Hadoop Cloud U of Chicago U of Florida Hadoop cloud Purdue Papers: “Sky Computing”, by K. Keahey, A. Matsunaga, M. Tsugawa, J. Fortes. Submitted to IEEE Internet Computing. “CloudBLAST: Combining MapReduce and Virtualization on Distributed Resources for Bioinformatics Applications” by A. Matsunaga, M. Tsugawa and J. Fortes. eScience 2008. 3/2/09 The Nimbus Toolkit: http//workspace.globus.org Thoughts… Science-driven cloud computing How does IaaS fit into our world? Combine with what we have (grid computing) Explore new potential Interoperability Academic vs commercial resources Standards: “rough consensus & working code” Importance of open source Drive requirements into the infrastructure, customize Drive the development of standards 3/2/09 The Nimbus Toolkit: http//workspace.globus.org .