CMS Remote Operations

• Jargon 101 • Where In The World • Not All Colliders Are The Same • Compact Is Relative • It’s A Small World • Parlez-Vous Français • LHC@FNAL

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Jargon 101

FNAL: Fermi National Accelerator Laboratory CERN: European Organization for Nuclear Research (Conseil Européen pour la Recherche Nucléaire) LHC: LEP: Large Electron Positron CMS: ATLAS: A Toroidal LHC Aparatus LPC: LHC Physics Center HEP: High Energy Physics LARP: LHC Accelerator Research Program ROC: Remote Operations Center WBM: Web-Based Monitoring MTCC: Magnet Test and Cosmic Challenge CSA07: Computing, Software, and Analysis 2007 http://www.uscms.org/roc/cms_glossary.html 16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 2 Where In The World

Chicago to Geneva ~ 4000 miles

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 3 CERN

Remember this distance. I’ll come Geneva back to it later. Airport

CERN Campus

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 4 Fermilab & CERN Campuses

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 5 The LHC Accelerator

The LHC tunnel is buried 50 to 175 m underground, and straddles the Swiss and French borders on the outskirts of Geneva. Proton-Proton collisions at an energy of 7 TeV per beam. First high-energy collisions are expected in mid-2008.

http://lhc.web.cern.ch/lhc/ 16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 6 LHC vs Energy: 14 TeV = 7 x Tevatron Length: 27 km = 4 x Tevatron Magnetic Field: 8.3 T = 2 x Tevatron Beam Energy: 350 MJ = 250 x Tevatron Bunch Collisions: 40 MHz = 20 x Tevatron Instantaneous Luminosity = 60 x Tevatron # of Collisions in an event = 10 x Tevatron Data Rate: 1 Terabyte / sec = 50 x Tevatron # of Detector Channels: 100 M = 100 x Tevatron # of Scientists (~2500/expt) = 3 x Tevatron

300 ft

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 7 Tevatron vs LHC Status

Tevatron LHC Aug-Oct 2006 Cosmic Ray Run 2000 (above ground) Engineering Run Oct. 2000 May-Nov 2007 with Partial Detectors (below ground) Detector Completion Mar 2001 Dec 2007 Spring 2008 Commissioning Run Apr 2001 - Feb 2002 (up to 7 TeV) Beginning of Physics Run Feb 2002 Summer 2008

• Time-in of electronics across all detector and trigger subsystems • Commission beam loss monitors • Calibration and alignment of each system • Establish stable detector configuration and trigger criteria

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 8 LHC Accelerator Schedule

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 9 CMS Basic Parameters

Parameter Value Bunch-crossing frequency 40 MHz Avg # of collisions / bunch-crossing 20 “interaction rate” ~109 Level-1 trigger rate 100 kHz Average event size 1 Mb Event builder bandwidth 100 Gb/s Event filter computing power 106 Si95 Event rate saved to mass storage 100 Hz Data production 10 Tb/day

Sub-Detector Number of channels Pixels 66 x 106 Silicon microstrips 11.4 x 106 ECAL crystals 0.076 x 106 Preshower strips 0.137 x 106 HCAL 0.01 x 106 Muon chambers 0.576 x 106 TOTAL 78.2 x 106

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 10 CMS Detector

http://cmsinfo.cern.ch/outreach/CMSdocuments/DetectorDrawings/Slice/CMS_Slice.swf 16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 11 Webcam View: Lowering Detector into Cavern on 19 Jan 2007

Several CMS webcams: http://www.uscms.org/roc/webcams.html 16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 12 CMS: The Countries

2000+ scientists and engineers 155 institutes 37 countries Belgium http://cms.cern.ch/ Austria Bulgaria

USA Oct 2006 CERN Finland France Germany Russia Greece Uzbekistan Ukraine Hungary Slovak Republic Italy Georgia Belarus UK Poland Armenia Turkey Portugal India Spain Pakistan China Estonia Switzerland Korea Cyprus CroatiaChina (Taiwan)

Ordered by size: USA (525 collaborators), Italy (398), Russia (326), CERN (204), France (146), UK (117), Germany (116)

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 13 US CMS Collaboration

49 Institutions

By size (physicists): FNAL: 58, Florida State: 21, UCLA: 15, UC Davis: 13, MIT: 13, Rochester: 13, Rutgers: 11

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 14 HEP Remote Operations • With the growth of large international collaborations in HEP, the need to participate in daily operations from remote locations has increased. – CDF and DZero at Tevatron have ~600 collaborators. – CMS and ATLAS at LHC will have >2000. • Remote monitoring of HEP experiments is nothing new. – The World Wide Web was invented at CERN in 1989, and the first web server came online at Stanford Linear Accelerator in Dec 1991. – The first web browser, Mosaic, was released in 1993. • Remote operations is the next step, to enable collaborators to participate in operations from anywhere in the world. Some of the goals include: – Secure access to data, devices, logbooks, monitoring information – Safeguards so actions do not jeopardize or interfere with operations – Collaborative tools for effective remote participation in shift activities – Remote shifts to streamline operations.

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 15 It’s A Small World

• We use technology every day to reduce the scale in distance and time to control and monitor many aspects of our lives including financial, social, health and entertainment. Some examples: – Checking your Gmail account from a laptop in Starbucks – Navigating through the Comcast On-Demand menu to select a new movie – Receiving an eBay wireless alert that someone has just outbid you – Calling Dell customer support at 2:35 am & reaching an agent in India – Paying your credit card bills from your online Chase bank account – Using On-Star to get your door unlocked remotely (keys are inside!) – Seeing an AMBER Alert on an electronic billboard on the way home – Tracing your genealogy to Slovenia on Ancestry.com – Finding a new apartment in Geneva, Switzerland on Craig’s List – Updating MySpace page with latest photos and news of the day – Researching Zillow database before putting your house on the market – Transfering your prescription to the nearest Walgreens while on vacation • Without realizing it, you perform some type of a remote operation every day, often with similar technology used by high energy physicists

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 16 Real World vs HEP • The internet and the web browser interface permits us to reach an instant audience from 1 to many with web-logs and social networking sites like MySpace and Facebook. These are very similar to HEP electronic logbooks which include text, tables, photos and image files, and to Wiki pages used throughout CMS to document most everything. • OnDemand Menus ⇔ Video Conferencing Menus – Interactive, Remote controls. • Secure Banking ⇔ Secure Web access to databases – Password protected. Data Integrity. • Customer Service ⇔ 24/7 helpdesk. – Computing systems, power, safety. Shift operations. • Automation. Email alerts. Web page updates. • Windows XP. Linux. Macs. Office. Oracle. Java. iTunes. • Multinational Corporation ⇔ CERN & CMS headquarters. – Fermilab base of US operations.

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 17 Parlez-Vous Francais • There are quite a number of human factors for establishing a remote operations center – Logistics: CERN does not have the infrastructure to provide office space, lodging, cafeteria services, travel, etc. if even one-third of all the various collaborations wanted to work on-site. – Language: Everyone speaks English at CERN, but you (and your family) need to learn some French if you want to buy anything or have a social life at night or on weekends. – Family: Many scientists have children in school, a working spouse, a mortage, two cars, etc., so they do not want relocate to CERN. – Cultural: TV and radio is not in English. Stores do not sell your favorite products, and are not open 24/7 or Sundays. All of your closest friends will remain back home. Most of your personal belongings will need to be put in storage. – Financial: The cost of living in Geneva is significantly higher than most major cities. Travel to and from Geneva is also a factor.

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 18 LHC@FNAL ROC Features • 4 CERN-style consoles (8 workstations) shared by CMS & LHC scientists • 4 Projectors to share content within the ROC or to remote participants • Videoconferencing installed for two consoles • Webcams for remote viewing of ROC • Secure keycard access to the ROC from Atrium and 1East Mtg Room • Secure network for console PCs – Dedicated subnet, dedicated router w/Access Control Lists to restrict • HD viewing of the ROC, and HD display access. capabilities in the ROC • 12-minute video essay displayed on • Secure group login capability for the large “Public Display” used by consoles, with persistent console docents from the Education sessions Department to explain CMS and LHC – Allows multiple users to share common to tour groups console settings. • Telephone lines share common number. • High Definition videoconferencing International services enabled. system for conference room • Access to LHC Physics Center computing resources.

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 19 CMS ROC Shift Activities The first group to use the ROC for shifts was the CMS Tier-1 computing administration operations team which was during weekday business hours. • Responsible for FNAL Tier-1 resources (~ 50% of CMS computing in U.S.) • Provides central support for the several university-based Tier-2 centers The first detector group to use the ROC for shifts was the silicon tracker. •Coordinated effort with the silicon tracker group & CMS ROC people working together. •The remote shift operation at the ROC involved about 15 people from several different institutions from Feb – Jun 2007 •Remote Monitoring included • Data Quality • Data transfer & bookkeeping • Event display • Detector Control System Current: CMS detector and trigger commissioning during the summer 2007.

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 20 Remote Operations for LHC

• Training prior to stays at CCC at CERN CERN • Remote participation in studies • ‘Service after the sale’: to support accelerator components built in the U.S. • Access to monitoring information • Software development for LHC controls system (LAFS) • LARP: The US LHC Accelerator Research Program (LARP) consists of four US laboratories, BNL, FNAL, LBNL and SLAC, who collaborate with CERN on the LHC. • The LARP program enables U.S. accelerator specialists to take an active and important role in the LHC accelerator during its commissioning and operations,CCC and to be a major collaborator in LHC performance upgrades. 16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 21 Remote Operations at CERN • Remember Slide 4? CMS detector site & control room is 30 minute drive from CERN campus & main CMS office buildings. – The general CMS population needs a Remote Operations Center on the CERN campus as the main CMS control room is too small & inconvenient. No cafeteria, no meeting space, no office space, etc. • A CMS Centre is under construction and will include: – Workspace for 25-50 people (300 sq meters). – Office space for 250 people. Priority for CMS Centre users working on Data Quality, Detector Experts, Computing Operations, etc. – Meeting Rooms & an Auditorium. – Outreach displays. • Construction should begin this summer. • LHC@FNAL ROC was used as a model for the CMS Centre. – Computing & Networking needs are similar. We are using the same consoles. CMS Centre will be 3-4 times larger.

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 22 Summary • Remote operations is the next step in high energy physics, enabling collaborators to participate in the experiment from anywhere in the world. – With secure access to data, safeguards to protect systems and effective collaborative tools, remote shifts can be performed from any time or place. – The cost of travel and/or the stress of relocation is mediated by remote operations. • Fermilab has built the LHC@FNAL Remote Operations Center, which is shared by scientists and engineers working on the LHC and CMS. • For the LHC it provides a means to participate remotely in LHC studies, access to monitoring information, a training facility, and supports the collaborative development of software for the LHC controls system. • For CMS it provides a location (in a U.S. time zone) for performing remote sub- detector commissioning and operations shifts, and Tier-1 grid monitoring shifts. • Plans for using the ROC in the summer and fall of 2007: - Trigger commissioning, Global Run & Commissioning and MTCC-III [CMS] - Data operations for CSA07 [Tier-1] - Synchotron Proton Source commissioning [LARP], - LHC software development [LAFS] - Education and Outreach.

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 23 CMS: A Worldwide Adventure

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 24 Backup Slides

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 25 LHC Accelerator Schedule

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 26 Wilson Hall 1st Floor Layout

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 27 LHC@FNAL Layout

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 28 CMS Collaboration

See http://cmsinfo.cern.ch/

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 29 Accelerator Challenges

Tevatron team in LHC commissioning

Tevatron

LHC

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - FermilabLHC 30 CMS Trigger & DAQ

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 31 Assignment of Responsibilities

CMS Control Room Remote Operations Centers

CMS Sub- Responsibilities for sub- Responsibilities for typical sub-detector: Detector detector: • coordinate sub-detector offline operations • ensure safe operation of sub- • detailed sub-detector data-quality monitoring detector and monitoring • liaise with counterpart in CMS Control Room hardware • run dedicated jobs on express-line and • take calibration data calibration streams • provide constants to the High • provide updated constants for High Level Level Trigger (HLT) Trigger (HLT) and Tier-0 processing of data • check real-time data-quality monitoring for the sub-detector

CMS Global • monitor detector, trigger, and • real-time monitoring of detector, trigger, and Run & Data data acquisition status data acquisition status (“mirrored displays”) Quality • monitor global data-quality monitoring results • monitor online data-quality from offline processing of data Monitoring monitoring results • collate run-quality information from all • communicate issues of data detectors quality to other shift personnel • coordinate preparation of good/bad run lists in the CMS Control Room • coordinate (with computing operations) scheduling of Tier-0 bulk processing of events • liaison with CMS Centre, for • check data quality after Tier-0 bulk processing example to request further • liaise with offsite remote centers and experts studies of specific problems

16 July 2007 - QuarkNet Fellows Teacher Outreach - Alan L. Stone - Fermilab 32