Icecube Project Monthly Report s5

IceCube Project Monthly Report June 2008

Accomplishments  The on-ice Integrated Master Schedule with critical dates and activities managed by both Raytheon Polar Services Company and the UW is set. The schedule begins drilling on December 1, 2008, and currently provides for the installation of 20 strings and 19 IceTop stations. Very recent correspondence from the NSF IceCube Project Officer indicates that IceCube installation plans will need to be reduced due to a reduction in flights available to the project for cargo, fuel, and passengers. The impacts are being evaluated.  DOM production at all sites and cable production is progressing on schedule.  The IceCube data systems acquired 3.3 billion events with an overall uptime fraction of 98.6%.  The upgrade of the Data Warehouse to 200 TeraBytes of disk and an upgraded HSM tape library system is underway with purchases authorized this month.  Work continued in June on upgrading the Processing and Filter (PnF) server for higher data throughput that will accommodate a full 80-plus string detector when completed.  Simulation production for the anticipated 40-string IceCube detector (IC-40) is in progress using IceSim Version 2.2. This is well ahead of last year’s IC-22 simulation production.

IceCube Project Baseline (M$) Initial In-Ice Strings & IceTop Tanks Installed Jan-2005 Initial Operational Capability Mar-2007 Project Completion & Closeout Sep-2011 Total Project Cost $275.3 300 Value of Foreign Contributions $33.2 NSF Funding $242.1 273.4 275.0 275.3 Remaining Contingency $10.9 260.5 264.4 Contingency as % of Remaining Work 28.3% 250 Strings installed 40 235.3 231.6 Performance and Cost through: 31-May-08

199.4 200

S FULL OPERATIONS N O

I & DATA ANALYSIS L

L CAPABILITY I M

I 150 140.1 INITIAL OPERATIONS & DATA ANALYSIS CAPABILITY S R A

L FULL SCALE DOM PRODUCTION L O D 100 85.7 DETERMINE FULL DEPLOYMENT RATE

___Total Funds (US & NonUS) ___Original Budget (BCWS) 50 39.5 DEPLOY INITIAL STRINGS & TANKS ___Current Budget (BCWS) 85.2 % ___Earned Value (BCWP) 85.5 % 15.0 COMPLETE EHWD INTEGRATION & TEST ___Actual Cost (ACWP) 85.1 %

0 FY02 FY03 FY04 FY05 FY06 FY07 FY08 FY09 FY10 FY11 FISCAL YEAR Cost and Schedule Performance – The project is 85.5% complete versus the plan of 85.2% complete, as measured using earned value techniques. The contingency % of the remaining work is 28.3%. IceCube Neutrino Observatory Cost Schedule Status Report Reporting Period Ending: 5/31/2008 Note 1 At Completion Cumulative (AY K$) Note 4 Complete (%) Budgeted Cost 2 Actual Cost Variance Risk ContingencyNote5 Work Work of Work Budgeted Actl OBS Structure L2 Scheduled Performed Performed Schedule Cost Assigned % ETC AY $s Sched Perf Cost Project Support 22,833.2 22,833.2 22,792.6 0.0 40.6 116.4 3.1% 26,832.6 85.1% 85.1% 84.9% Implementation 34,668.1 34,668.1 34,523.0 0.0 145.1 1,865.4 14.1% 46,939.7 73.9% 73.9% 73.5% Instrumentation 66,216.7 66,730.8 66,739.9 514.1 -9.0 328.6 6.9% 71,498.0 92.6% 93.3% 93.3% Data Acquisition 33,236.8 33,236.8 33,197.7 0.0 39.1 86.2 14.4% 33,798.3 98.3% 98.3% 98.2% Data Systems 23,128.4 23,128.4 23,204.1 0.0 -75.7 88.0 2.5% 26,666.6 86.7% 86.7% 87.0% Detector Comm. & 18,618.0 18,618.0 18,605.8 0.0 12.2 214.6 10.0% 20,754.5 89.7% 89.7% 89.6% Verification Pre Operations 573.4 573.4 527.8 0.0 45.6 0.0 0.0% 924.2 62.0% 62.0% 57.1%

Subtotal 199,274.5 199,788.7 199,590.8 514.1 197.8 2,699.2 9.5% 227,414.0 87.6% 87.9% 87.8%

RPSC SUPPORT 25,200.2 25,381.3 24,706.2 181.1 675.2 593.5 5.4% 35,745.8 70.5% 71.0% 69.1% NSF 849.9 849.9 849.9 0.0 0.0 24.8 6.0% 1,263.0 67.3% 67.3% 67.3%

Total 225,324.6 226,019.8 225,146.9 695.3 873.0 3,317.5 8.3% 264,422.7 85.2% 85.5% 85.1%

CONTINGENCY Notes 3 10,870.7

IceCube Total Note 2 225,324.6 226,019.8 225,146.9 695.3 873.0 3,317.5 8.3% 275,293.5 85.2% 85.5% 85.1%

Notes: 1 Incorporates approved baseline changes. 2 Total Budget at Completion includes non-US contributions3,522 K over the amount in the post Hartill III baseline. 3 Budgeted contingency is: 28.3% of the Budgeted cost of work remaining. 4 The BAC (Budget At Completion), reflects PY7-10 detailed planning Baseline Re-Plan. 5 Contingency is assigned based on the ETC, a bottom-up risk assessment, management judgement, and cost constraints. The favorable total $873k cost variance at the end of May 2008 is driven almost entirely by a $675k variance in RPSC Support. Lagging seasonal costs that created a large variance last month have now been realized. The remaining positive cost variance results from lower than planned labor charges in RPSC Division Management, and FY08 Actual costs for On-Ice Support that were less than planned.

5/31/08 Change Log - IceCube Total Project Budget Baseline ($K) Allocated Budgeted Cost Contingency % Total Allocated Contingency No. Description Budget of Work of Remaining Baseline Budget Budget Change Remaining Work NA Status as of April 2008 275,293 264,423 0 10,871 40,732 26.7%

NA Status as of May 2008 275,293 264,423 0 10,871 38,403 28.3% IceCube Contingency: % of Remaining Work 40%

30%   28.3% O   

20% Contingency Usage Implementation 62% Annual Baseline Re-Plan 2005, 06, 07, 08 Instrumentation 26%  10% DAQ 10% O Additional IceTop Stations + Installation Season RPSC 9% Instrumentation for >75 strings Project Support -11% Others 4% 0% Feb- May- Aug- Nov- Feb- May- Aug- Nov- Feb- May- Aug- Nov- Feb- May- Aug- Nov- Feb- May- 04 04 04 04 05 05 05 05 06 06 06 06 07 07 07 07 08 08

Risk Assessment & Potential Contingency Adjustments

Item Estimate ($K) 1. Assign contingency to mitigate technical, cost and schedule risks associated with the defined scope of work. Risk applied at WBS-Level 4 to assess appropriate $3,318K contingency as a percent of the cost of work remaining.

2. Current RPSC estimate of base cost to launch the 2010/11 drilling season. $920K 3. Pre-Operations Exposure for Upgrading Data Storage - 1st phase June 2008: Upgrade the Data Storage and South Pole Systems to meet the collaboration's $870K needs. 4. Pre-Operations Exposure for Upgrading Data Storage - 2nd phase August 2008: Upgrade the Data Storage and South Pole Systems to meet the collaboration's $700K needs. 5. Instrumentation critical path materials for up to 86 strings (Purchase 5 Surface to $1,350K DOM cable systems, build ~100 DOMs) 6. Current RPSC estimate of base cost to support the installation of 6 strings during $1,340K the 2010/11 drilling season. 7. The cost to retro IceCube equipment/materials from the South Pole at the end of $825K the project in FY2012, was not included in the baseline budget. 8. Assign contingency against major fuel price increases beyond normal inflation. $500K Total $9,823K Available Contingency as of May 31, 2008 $10,871K

Fuel Cost Increase Impact Analysis - IceCube actual costs are sensitive to increasing energy costs. The Antarctic Program’s “pump price” paid for fuel that will be used during the FY2009 season was $2.85 per gallon (FY2009 fuel was already shipped). The price is expected to substantially increase for FY2010 and beyond. The project is evaluating the potential impact of higher fuel costs in the last two drilling seasons (FY2010 and FY2011) and for subsequent retro cargo transportIceCube Fuel (FY2012). Cost Increase ThisImpact includes Analysis fuel for drilling,IceCube cargo Fuel transport, Cost Increase personnel Impact Analysis at Pole, and $600 K $600 K heavy equipment use. FY10 20%, 2.5%, 2.5% 20% each year 510 510 $500 K FY10 30%, 2.5%, 2.5% $500 K 30% each year 382 424 40% each year $400 K 382 FY10 40%, 2.5%, 2.5% $400 K 305 $300 K 255 $300 K 255 192 259 $200 K $200 K 195 147 178 108 $100 K 70 $100 K 102 54 $0 K 39 $0 K FY10 FY11 FY12 FY10 FY11 FY12 FY09 Baseline fuel rates: 2.85 $/gal at the pump, 11.73 $/gal at the pole FY09 Baseline fuel rates: 2.85 $/gal at the pump, 11.73 $/gal at the pole The project previously included $180K for potential fuel cost increases in the WBS-Level 4 baseline risk assessment, Item #1 in the risk table above. We are now also including an additional $500K potential contingency adjustment, for the extraordinary current pricing. The graphs above show the total cost exposure as a result of potential fuel cost increase. The graph on the left shows the impact of a 20 – 40% increase each year and the graph on the right shows a one time increase in FY2010 of up 20 – 40% and additional 2.5% inflation in FY2011 and FY2012.

Drill Operation and Installation – A new main drill cable was ordered. Two cables that were brought back from the South Pole were found to be acceptable for refurbishment and reuse. One of the two cables will be refurbished by reusing the core and strength member. The other cable will be kept as a reserve for redesign and refurbishment in the event problems continue to be encountered. A cable prototype is currently being tested at the UW-Madison Physical Sciences Laboratory for possible redesign improvements to the new cable and refurbished cable.

The on-ice Integrated Master Schedule with critical dates and activities for Raytheon and the UW has been set. The schedule was based upon on the current flight plan.

Digital Optical Module and Cable Production Status and Plans - DOM production is in progress at all 3 sites and is progressing smoothly. Surface to DOM cable production is on track. Procurement plans have been developed for instrumentation (primarily DOM hardware and cables) to restore IceCube to the originally planned 80 string configuration. This will include 15 surface cables and 6 surface-to-DOM cables, and associated breakouts and connectors. IceCube DOM Integration PY7 (April, 2008 to October, 2008) - Plan vs. Actual Plan to IC75+ with existing materials on hand with exceptions 4/18/08 (see Jonathan's email)

DOMs ready for DFL @ PSL - cumulative plan

700 DOMs ready for DFL @ DESY - cumulative plan DOMs ready for DFL @ Swed - cumulative plan

DOMs ready for DFL - cumulative plan all sites 600 DOMs ready for DFL - cumulative actual all sites L F D

500 y d a e r

M 400 O D

i 300 t n a u Q 200

0 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 / / / / / / / / / / / / / / / / / / / / / / / / / / / / 5 2 9 6 3 0 7 4 1 7 4 1 8 5 2 9 6 2 9 6 3 0 6 3 0 7 4 1 / / / / / / / / 1 1 2 1 1 2 3 1 2 2 1 1 2 1 2 3 1 2 2 1 4 / / / 5 / / / / 6 / / / 7 / / / 8 8 / / / 9 / / / 0 / 4 4 4 5 5 5 5 6 6 6 7 7 7 8 8 8 9 9 9 1 0 1 Week ending

Detector Commissioning and Verification – Verification continues to run since the start of IC40 data taking. A handful of DOMs have been found with problems that have been referred to hardware experts: In one DOM, the baseline appears to have a slope; in about 20 DOMs, the position of the single photoelectron peak exhibits abnormal variation with time. The first version of a new webpage to display the verification information and make it easier for verification experts, data analyzers and shift takers to see problems has been implemented. It continues to be tested and used by the verification group. The verification group expects to begin generating alarms based on verification information, using the IceCube Live interface for this purpose. The alarms will consist of emails sent to members of the verification group, who will then decide how to respond.

Calibration and Monitoring: A discrepancy in the channel 1 and channel 2 (different gain inputs to the ATWD), has been observed in some DOMs. There may be a stronger than expected dependence on frequency for the channel 2 gain. This is under investigation. Calibration data-taking using various in-situ light sources to connect the AMANDA ice property measurements with IceCube measurements (using AMANDA string 18 flashers), and measure the bulk ice properties using DOM flasher board LEDs and the Standard Candle lasers, are all in the planning or pilot run stage. Standard Candle runs have also been proposed to help understand the detector response to the large light depositions expected from extremely high energy events.

Data Acquisition Software - During the period June 1, 2008 through June 30, 2008, 3.3 billion triggered events were collected from the IceCube DAQ with an overall up time fraction of 98.6%. The day-by-day chart of uptime fraction is graphed in the following figure: The dip on June 22nd down to 94% uptime was due to a crash of a DOMHub, which took the detector out of service for approximately one-hour. No new software releases were issued in June. The developers have concentrated on the following issues/features:

 IceTop calibration trigger feature request  Sub-threshold bug #2 fix  Implementation of signal simulation/data challenge for supernova subsystem

All are scheduled for the next release planned for mid-July. IceCube Maintenance and Operations

Detector Up Time 98.6% IceCube Clean Run Up-Time not including AMANDA Array 96.5% Unscheduled Downtime 0.2% Events from DAQ 3.28 billion

IceCube Detector Operation for June 2008

90% Unscheduled Downtime 80% Maintainance

y 70% C&V a D

Special Tests / commiss. of new strigs f 60% o

DOM Calibration e

a Monthly Benchmark/Calibrations t

n Partial IceCube + AMANDA e 40% c

r Partial IceCube e

P 30% AMANDA Only 20% IceCube Only IceCube+AMANDA 10%

0% 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 Day Data taking during June 2008 achieved an overall 98.6% uptime. The in-ice clean run uptime achieved a 96.5%.

Data Systems

Data Handling - South Pole systems in the IceCube Laboratory (ICL) began IC-40 physics operation in April. Operations have continued smoothly through June. The figure below shows the daily satellite data transfer rates. Work in June focused on completion of requirements and purchasing of equipment for shipment and installation at the South Pole ICL to accommodate the new strings and larger processing requirements for higher data rates.

There is also ongoing work in pre-operations for this category to enhance the Data Warehouse with a major upgrade of 200 TB of disk and an upgraded HSM tape library system. Major purchasing began in June for this data warehouse upgrade. Filtering, Software & Database - Filtering of IC40 data began on April 14, 2008 representing the full transition from IC-22 to IC-40. The Joint-Event Builder and Processing and Filter server (JEB/PnF) merging of IceCube-40 and Amanda TWR data streams continues at South Pole sending filtered data sets of about 35 GB/day over the satellite. Work continued in June on upgrading the PnF server with higher data throughput, which will accommodate the full 80 plus string detector when completed. The upgraded PnF will also include better monitoring and integration with the IceCube Live control system to be deployed this year.

Preparations and testing continue for offline processing on the IC-40 data as it flows north over the satellite. Routine processing of L0/L1 is finalizing with the Level 2 processing requirements and sign off by the physics working group to be completed soon. This schedule is nearly six months ahead of last year’s IC-22 offline processing. Preparations and negotiations with the European KM3NET collaboration for use of our software framework IceTray continues. There will be a working meeting with the two collaboration’s technical people in early August.

Simulation - Mass simulation production with IceSim Version 2.1 to produce large background and signal Monte Carlo datasets for the physics working groups to prepare for IC-22 physics analysis has been completed. Mass production for IC-40 has begun, with IceSim Version 2.2, well ahead of last year’s IC-22 simulation production.

Education and Outreach – Two high school students participated in three-week ITA internships in the IceCube IT department in Madison. ITA is an innovative, 4-year pre-college technology access and training program for talented students of color and economically disadvantaged students attending Madison Public Schools. Each year, ITA competitively recruits 30 students in their final semester of 8th grade to participate in the program. Selected students receive four years of intensive training in preparation for high tech, IT related careers; in addition to intensive academic support in preparation for competitive University admissions and study. The high school junior or senior ITA interns with IceCube worked with the webmaster and performed IT support duties.

Quality Assurance and Safety – No incidents to report.

The monthly reports are posted at IceCube Monthly Reports.

Meetings and Events Driller Training @ UW-Madison July 28-August 8, 2008 International Oversight & Financial Group Meeting TBD, September 2008 Fall Collaboration Meeting @ Utrecht, Netherlands September 15-19, 2008