Icecube Project Monthly Report s4

IceCube Project Monthly Report - December 2008 Accomplishments

 Drilling and string installation at the South Pole is about one week ahead of the schedule.

 Ten strings were successfully installed by the end of December 2008 and a total of 16 strings were deployed by January 15th. The deep core prototype string was the 16th string installed.

 IceTop installation work for the current season is complete, tanks are filled with water, and the controlled freeze of the water in the tanks is underway.

 Additional computing hardware was installed in the IceCube Laboratory (ICL) at the South Pole for DAQ, processing and filtering software, and calibration and verification systems. A tape based storage system was installed at the IceCube data center in Madison.

 IC40-string operations during December 2008 achieved an overall uptime of 97.3%.

 The IceSim MonteCarlo event simulation program is well advanced and further development is limited to the continuous improvement of the program.

 An IceCube collaborator was evacuated from South Pole for a non-injury condition and is recovering off-ice. There are no work related injuries this season.

Drilling at the location of the 5th hole of the season, the 45th overall, and concurrent deployment at the location of the 4th hole of the season. Photo on December 13, 2008 by Tom Gaisser Cost and Schedule Performance - The project is 88.8% complete versus the plan of 88.7% complete, as measured using earned value techniques. Remaining contingency is $8.6M and the contingency as a percentage of remaining work is 28.9%. The contingency continues to be carefully managed to ensure the successful completion of the MREFC project.

IceCube Project Baseline (M$) Initial In-Ice Strings & IceTop Tanks Installed Jan-2005 Initial Operational Capability Mar-2007 Project Completion & Closeout Sep-2011 300 Total Project Cost $275.3 Value of Foreign Contributions $33.2 275.3 NSF Funding $242.1 273.4 275.0 Remaining Contingency $8.6 260.8 266.6 Contingency as % of Remaining Work 28.9% 250 Strings installed 40 235.3 231.6 Performance and Cost through: 30-Nov-08

199.4 200

S FULL OPERATIONS N O

I & DATA ANALYSIS L

L CAPABILITY I M

I 150 140.1 INITIAL OPERATIONS & DATA ANALYSIS CAPABILITY S R A

L FULL SCALE DOM PRODUCTION L O D 100 85.7 DETERMINE FULL DEPLOYMENT RATE

___Total Funds (US & NonUS) ___Original Budget (BCWS) 50 39.5 DEPLOY INITIAL STRINGS & TANKS ___Current Budget (BCWS) 88.7 % ___Earned Value (BCWP) 88.8 % 15.0 COMPLETE EHWD INTEGRATION & TEST ___Actual Cost (ACWP) 88.1 %

0 FY02 FY03 FY04 FY05 FY06 FY07 FY08 FY09 FY10 FY11 FISCAL YEAR

IceCube Neutrino Observatory Cost Schedule Status Report Reporting Period Ending: 11/30/08 Note 1 At Completion Cumulative (AY K$) Note 4 Complete (%) Budgeted Cost 2 Actual Cost Variance Risk ContingencyNote5 Work Work of Work Budgeted Actl OBS Structure L2 Scheduled Performed Performed Schedule Cost Assigned % ETC AY $s Sched Perf Cost Project Support 23,847.0 23,847.0 23,775.0 0.0 72.1 98.5 3.2% 26,832.6 88.9% 88.9% 88.6% Implementation 37,330.2 37,330.2 36,707.4 0.0 622.8 1,366.2 13.4% 46,939.7 79.5% 79.5% 78.2% Instrumentation 69,826.8 69,856.9 69,818.1 30.1 38.7 107.5 6.4% 71,498.0 97.7% 97.7% 97.7% Data Acquisition 33,558.4 33,558.4 33,451.3 0.0 107.1 44.4 12.8% 33,798.3 99.3% 99.3% 99.0% Data Systems 24,131.3 24,131.3 24,224.2 0.0 -92.9 54.9 2.2% 26,666.6 90.5% 90.5% 90.8% Detector Comm. & 19,334.4 19,334.4 19,217.3 0.0 117.0 121.9 7.9% 20,761.2 93.1% 93.1% 92.6% Verification Pre Operations 2,035.7 2,035.7 1,895.5 0.0 140.1 0.0 0.0% 2,212.9 92.0% 92.0% 85.7%

Subtotal 210,063.9 210,094.0 209,088.9 30.1 1,005.0 1,793.5 9.1% 228,709.4 91.8% 91.9% 91.4%

RPSC SUPPORT 25,516.7 25,683.6 25,026.3 166.9 657.3 591.2 5.1% 36,672.2 69.6% 70.0% 68.2% NSF 920.4 920.4 920.4 0.0 0.0 20.6 6.0% 1,263.0 72.9% 72.9% 72.9%

Total 236,501.0 236,698.0 235,035.7 197.0 1,662.3 2,405.3 7.6% 266,644.5 88.7% 88.8% 88.1%

CONTINGENCY Notes 3 8,648.9

IceCube Total Note 2 236,501.0 236,698.0 235,035.7 197.0 1,662.3 2,405.3 7.6% 275,293.5 88.7% 88.8% 88.1%

Notes: 1 Incorporates approved baseline changes. 2 Total Budget at Completion includes non-US contributions3,522 K over the amount in the post Hartill III baseline. 3 Budgeted contingency is: 28.9% of the Budgeted cost of work remaining. 4 The BAC (Budget At Completion), reflects PY7-10 detailed planning Baseline Re-Plan. 5 Contingency is assigned based on the ETC, a bottom-up risk assessment, management judgement, and cost constraints.

The earned value schedule variance at the end of November 2008 is a favorable $197K, slightly reduced from the variance last month. The variance is mostly related to the RPSC’s FY2008 schedule performance variance.

The earned value cost variance at the end of November 2008 is a favorable $1,662K, an increase from the $1,459K reported at the end of last month. RPSC has a positive cost variance of $657K due to lower than planned labor and On-Ice support charges in FY2008. The $622K favorable variance in Implementation was the result of lower training costs than planned, lagging invoices, and the ramp down of some labor ahead of schedule.

Contingency Status and Plans The contingency % of remaining work increased from 27.5% to 28.9% during November. The PY8 (April 1, 2009 and beyond) planning is nearly complete and will result in about a $1M draw on contingency, less than in previous annual planning cycles.

11/30/08 IceCubeIceCube Total Contingency Project Budget % of Baseline Remaining - Change Work Log ($K) 40% No. Description Total Allocated Allocated Contingency Budgeted Contingency % Baseline Budget Budget Budget Cost of of Remaining   Change Work Work28.9% 30%  Remaining   NA Status as of 275,293 266,638 0 8,656 33,315 26.0% September 2008 20%NA Status as of October 275,293 266,638 0 8,656 31,461 27.5% 2008 Contingency Allocations Drilling, Installation, Logistics 58% $20,539 K CR0136 InstrumentationExtend Verification 24%275,293$8,445 K 266,645 Annual Planning7 of Estimate8,649 To Complete Raytheonat Clark-Atlanta (RPSC) Support U. 11% $3,881 K 10% Data Acquisition System 9% $3,339 K  Additional 10 IceTop Stations & Installation Season NA Pre-OperationsStatus as of 6%275,293$2,213 K 266,645 0 8,649 29,947 28.9%  Instrumentation for 5 Additional Strings ProjectNovember Support 2008 -10% -$3,536 K Others 1% $302 K  RPSC Support for FY11 season, Data Storage Increase Total 100% $35,183 K 0% Feb- May- Aug- Nov- Feb- May- Aug- Nov- Feb- May- Aug- Nov- Feb- May- Aug- Nov- Feb- May- Aug- Nov- 04 04 04 04 05 05 05 05 06 06 06 06 07 07 07 07 08 08 08 08 The cost of fuel for the FY2010 season is now estimated to be $130k less than planned since it was purchased in late December 2008 after oil prices dropped. The cost of fuel in FY2011, the final installation season, will be purchased in one year. As a result of the lower than planned FY2010 fuel cost and fuel consumption in FY2009 that is less than baseline plans, the “potential fuel cost exposure” of $600k was removed from the contingency table below. This will be reassessed during the coming year.

Risk Assessment & Potential Contingency Adjustments Item Estimate ($K) 1. Contingency likely to be required to mitigate technical, cost and schedule risks associated $2,405K with the approved scope of work. Risk assessments are made at WBS-Level 4 to determine the value of the risk exposure as a percent of the cost of work remaining. 2. Pre-Operations for additional enhancements to the data storage and network systems both at $500K the South Pole and in the Northern Hemisphere.

3. RPSC FY08 Closeout. The closeout is expected to return of RPSC authorized budget to -$400K IceCube contingency resulting from lower labor use and reduced support charges. 4. Annual estimate-to-complete for the remainder of the project for all elements of the work $1,000K scope except RPSC support. This planning process is nearly complete. 5. Estimate cost of the limited amount of remaining instrumentation for an 86 string detector, $1,350K 80 strings from the original proposal plus the six deep core strings financed by European partners. The additional instrumentation is five surface-to-DOM cables and ~100 DOMs. 6. Current RPSC estimate of base cost to support the installation of six strings during the $1,340K 2010/11 drilling season. The current RPSC baseline includes support for 80 strings. 7. The estimated cost to retro IceCube equipment/materials from the South Pole at the end of $825K the project in FY2012. 8. Potential cost exposure should a full FY2011 drilling season be required resulting in the $450K need to send UW personnel to the South Pole in FY2012 to dismantle the drill equipment/materials and prepare the equipment for retro out of South Pole. Total $7,470K Available Contingency as of Nov 30, 2008 $8,649K

Drill Operation and Installation – Drilling at the South Pole continues ahead of schedule. Very minor problems were encountered, but all have been resolved successfully. Firn drilling for the last hole of the season was completed the week of January 12, 2009. By the end of December 2008, seven days ahead of schedule, ten new strings were deployed. As of January 15th, a total of 16 strings were successfully deployed including the deep core prototype string. Efficiencies in drilling resulted in less fuel use than originally planned. NSF approved continuing drilling and installation of up to 19 strings. It is now anticipated that the 19th string will be installed by January 22nd. The drilling crews will then proceed with camp winterization and depart South Pole about one week ahead of schedule.

Digital Optical Module and Cable Production Status and Plans – DOM Integration ceased as planned at all production sites pending the delivery of materials in November and December. It will resume in January 2009 at PSL and in Sweden. DESY in Germany successfully completed all of their DOM Integration and Test activities on schedule. DOM production is not a critical path activity.

Detector Commissioning and Verification – Anna Franckowiak (Humboldt/Berlin) and Stephanie Hickford (Canterbury/Christchurch) were trained during this month and previous months for their upcoming trip to the Pole. They will be responsible for high-level commissioning of the new strings being deployed, and for taking the data so that we can verify the timing and calibrate the geometry of those strings. AMANDA will conclude operations at the end of the IC-40 run in March 2009.

Data Acquisition Hardware and Software – The IceCube clock system had a firmware bug because there was a leap second in a leap year which caused the clock to register two instead of one leap second and has caused an offset of 1 second relative to UT. The problem has been fixed but we are shopping around for another unit since this is the second time we have encountered a bug like this in the past 4 years.

IceCube Maintenance and Operations

Detector Up Time 97.3% IceCube Clean Run Up-Time not including AMANDA Array 94.5% Unscheduled Downtime 1.49% Events from DAQ 3.76 billion

Data taking during Dec 2008 achieved an overall 97.3% uptime. The in-ice clean run uptime was 94.5%, lower than typical but high considering the large amount of activities and special runs/configurations as the pole season unfolds. SPS is being maintained and upgraded and caused some loss of uptime.

On Dec 4th special runs were taken to develop and successfully test the Soft Local Coincidence pDAQ feature, planned for the next running cycle to begin in April. On Dec 9th and 12th data taking was interrupted for about five and three hours respectively by failure of multiple drives on a new server for the secondary event builder.

In the midnight between Dec 30 and 31st the IceCube GPS produced a wrong leap second. It was caused by the combination of the leap year (366 days in 2008) with the leap second that was accrued in the transition to the new year (which was indeed accrued in addition to the wrong additional one). As a result, on Dec 31st the AMANDA and IceCube clocks went out of phase and the overall detector rates increased. This issue is now resolved.

Data Handling – South Pole systems in the ICL began IC-40 physics operation in April 2008. Operations continued with systems operating relatively smoothly. The figure below shows the daily satellite data transfer rates.

The Data Handling work in December 2008 consisted of: 1) new hardware installed at South Pole for DAQ, processing and filtering, calibration and verification, and data handling and

archiving; 2) additional DOM Hubs installed to support strings and IceTop tanks installed in the 2008/2009 season; 3) uninterruptible power supplies added and configured at the Pole to support additional computing; 4) new HSM tape based storage system installed at IceCube Data Center. Performance tuning is ongoing; and, 5) Operating Systems upgraded on the SPS system.

Filtering, Software & Database – Filtering of IC40 data began on April 14, 2008 representing the full transition from IC-22 to IC-40 running. The Joint-Event Builder and Processing and Filter server (JEB/PnF) merging of IceCube-40 and Amanda TWR data streams continues at South Pole sending filtered data sets of about 35 GB/day or more over the satellite. The major work in online filtering and software systems included: 1) new JEB release supporting: TWR+I3 flashing, new IceTop MB readouts and a filtering for them; 2) new Database tools that simplify the loading of information at pole, and mirroring of the DB in the North. These replace the existing set and have more uniform interfaces across all scripts; and, 3) Icetray V3 development work moving forward nicely. Anticipated release of this final version for end of project year is on schedule. Offline filtering of the IC40 data made significant progress last month. Accomplishments include:  Finished standard processing script or for IC40 processing including all input from the physics working groups.  Release of the standard processing meta-project (Version 3.0.0) for IC40 processing.  Started mass production of high level reconstructions at the IceCube datacenter in UW.  IC40 data tapes arrived from South Pole. Preparation and tests for re-filtering the early month of the IC40 run to take advantage of improved calibrations.

Simulation – IceSim MonteCarlo event simulation program is fairly mature now, and further development is somewhat limited to improved performance and fine tuning. Some details include:

 The TWR simulation has been adjusted to the new DMADD thresholds measurements from the pole.  Significant work on cleaning up the PhotonicsService project, and added a more efficient calling sequence for use with photorec, following the findings and recommendation of the code review.  Planning to improve memory economy by merging the detector simulation modules. This will be a major change. Initial implementing of code has begun, with delivery for first testing in about a month.  There is also a plan to use the same threshold treatment for IceTop and InIce.

In addition to work on the simulation itself, there was significant progress on simulation production and filtering of simulation data. The main disk in the data warehouse /data/sim is read-only because it is full, and we cannot currently write data on it. This is due to the transition work in the date warehouse upgrade that is ongoing. We have temporarily used another available disk. We had planned to proceed with simulation data filtering rather than increase the volume of trigger level data files on disk for the moment. However, the work in December was severely hampered because of limited data availability during the date warehouse enhancements.

Education and Outreach – IceCube participated in the IPY's "Above the Poles" with a live event on December 4, 2008 (http://www.polartrec.com/live-from-ipy/archive). We also updated our IPY website for this date. Scientist, Mark Krasberg continued his Ice Stories Dispatches for the Exploratorium in December including a webcast from the South Pole on December 18, 2008 (http://icestories.exploratorium.edu/dispatches/index.php). Tom Gaisser, IceCube spokesperson, was interviewed on Discovery Channel Canada (Daily Planet) on December 22, 2008. When broadcast, the program will feature video of IceCube activities at the South Pole and an animation of the hot water drilling.

Quality Assurance and Safety – No significant issues to report. There are no reportable injuries so far this season. There are a few incident reports.

The monthly reports are posted at IceCube Monthly Reports. The On-Ice Weekly Construction reports are posted at On-Ice Weekly Construction Report.

Prince Albert of Monaco is signing a DOM on the 16th of the season, the 56th overall. This string is the 1st of the six Deep Core strings and was deployed on Wednesday, January 14th. Photo by: Sven Lidstrom.

Meetings and Events IceCube Software & Computing Review at UW-Madison March 24 – 25, 2009 Season Review Meeting at UW-Madison March 25, 2009 IceCube Maintenance & Operations Proposal Due April 1, 2009

Spring Collaboration Meeting at UW-Madison April 28 – May 2, 2009 NSF M&O Review at UW-Madison TBD Science Advisory Committee Meeting at UW-Madison May 20 – May 21, 2009 NSF Annual Review – Proposed Date June 3 – 4, 2009