Industrial Symbiosis in Heat Recovery Collaborations Between Data Centers and District Heating and Cooling Companies

DEGREE PROJECT IN THE FIELD OF TECHNOLOGY INDUSTRIAL ENGINEERING AND MANAGEMENT AND THE MAIN FIELD OF STUDY INDUSTRIAL MANAGEMENT, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2017

Industrial Symbiosis in Heat Recovery Collaborations between Data Centers and District Heating and Cooling Companies

JESSICA LIND

ERIK RUNDGREN

KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF INDUSTRIAL ENGINEERING AND MANAGEMENT

Industrial Symbiosis in Heat Recovery Collaborations between Data Centers and District Heating and Cooling Companies

Jessica Lind Erik Rundgren

Master of Science Thesis KTH School of Industrial Engineering and Management Energy Technology EGI_2017-0037 –MSC SE 100-44 Stockholm

Master of Science Thesis EGI_2017-0037 MSC

Industrial Symbiosis in Heat Recovery Collaborations between Data Centers and District Heating and Cooling Companies

Jessica Lind Erik Rundgren Approved Examiner Supervisor

Per Lundqvist Commissioner Contact person

Staffan Stymne Staffan Stymne Abstract Tough competition from local heating and cooling solutions fuels the pursuit of new cost- efficient heat sources for district energy companies. This master thesis explores the possibilities for a district heating and cooling company to integrate data centers for recovery of residual heat. Recovery of the residual heat from data centers into the aggregated heat demand of a district energy system can lead to less primary energy input for the two operations combined. The thesis is based on a single case study on Norrenergi, a district heating and cooling company in the Stockholm region, and nine semi-structured interviews with representatives from the data center industry. An industry interview compilation, case company interviews, and literature review were applied on a framework for industrial symbiosis. Synergies, enabling factors, and obstacles to successful cross-industry collaboration was evaluated and visualized in SWOT-analyses for four different examples of configuration for integration. The analysis shows that integration of data centers into district energy systems can provide synergies on all levels of industrial symbiosis: By-product reuse; Utility/infrastructure sharing; and joint provision of services. The data center market consists of diverse actors, with a range of business models and operational conditions. Some key obstacles to instigate collaboration are related to cultural differences, reliability concerns for district cooling, uncertainty in heat delivery potential from data centers, and data center reluctance to invest in equipment for heat recovery. To mitigate the obstacles, the energy companies should tailor the service offer for each collaboration on the diversified data center market. Residual heat from data centers should be recovered in the district cooling system, where possible. When district cooling is not accessible, energy companies should explore the possibility to invest in local heat pumps for recovery into the district heating supply flow. The thesis concludes that energy companies interested in heat recovery from data centers can apply a broad range of mitigation tools to facilitate collaborations.

Sammanfattning Hård konkurrens från lokala värme- och kyllösningar driver fjärrvärmeföretag att leta efter nya, billiga värmekällor. Det här examensarbetet har utforskat möjligheterna för ett fjärrvärme- och fjärrkylaföretag att integrera datacenter i sitt energisystem för återvinning av restvärme. Återvinning av restvärme från datacenter till det aggregerade värmebehovet hos ett fjärrvärme-och fjärrkylasystem kan leda till minskad total primärenergianvändning för de båda verksamheterna tillsammans. Examensarbetet baseras på en fallstudie hos Norrenergi, ett fjärrvärme- och fjärrkylaföretag i Stockholmsregionen, samt nio semi-strukturerade intervjuer med representanter från datacenterindustrin. Ett ramverk för industriell symbios applicerades på en sammanställning av intervjuerna från datacenterindustrin, intervjuerna på Norrenergi samt en litteraturstudie. Synergier, möjliggörande faktorer samt hinder för framgångsrikt, inter-industriellt samarbete utvärderades och visualiserades i SWOT-analyser för fyra olika integrationsmöjligheter. Analysen visar att integrationen av datacenter in i fjärrvärme-och fjärrkylasystem kan erbjuda synergier som korresponderar till alla nivåer av industriell symbios: återvinning av restprodukter, delad infrastruktur och gemensam tillgång till tjänster. Marknaden för datacenter består av olika aktörer med flera olika affärsmodeller och förutsättningar för verksamheten. Några av de främsta hindren för samarbeten mellan energiföretag och datacenter är relaterade till kulturella skillnader, oro över hur pålitligt fjärrkyla är, osäkerhet för potentialen för värmeleverans från datacenter och datacentrens ovilja att investera i utrustning för värmeåtervinning. För att överbrygga hindren behöver energiföretagen skräddarsy tjänsteerbjudandet till varje samarbete på den diversifierade datacentermarknaden. Restvärme från datacenter bör, om möjligt, återvinnas i fjärrkylasystemet. När fjärrkyla inte är tillgängligt bör energiföretag utforska möjligheten att investera i lokala värmepumpar för värmeåtervinning till fjärrvärmenätets framledning. Sammanfattningsvis kommer studien fram till att för energiföretag intresserade av värmeåtervinning från datacenter så finns det flertal verktyg för att främja samarbeten.

Abbreviations CapEx Capital Expenditure CHP Combined Heat and Power COP Coefficient of Performance CRAC Computer Room Air Conditioning CRAH Computer Room Air Handling DCaaS Data Center as a Service DES Distributed Energy System EIP Eco-Industrial Park GDP Gross Domestic Product GWh Giga Watt-hour HPC High Performance Computing ICT Information and Communications Technology IPSO Integrated Product and Service Offer KTH Kungliga Tekniska Högskolan kW kilo Watt MW Mega Watt OpEx Operational Expenditure PDC Parallel Data Center PUE Power Usage Effectiveness TCO Total Cost of Ownership TPA Third-Party Access

Table of Content 1 Background ...... 1 1.1 The District Heating Market ...... 1 1.2 Data Centers ...... 2 1.3 District Heating and Data Center Integration Projects ...... 2 1.4 Case Company ...... 2 1.5 Problematization ...... 3 1.6 Purpose ...... 3 1.7 Research Question ...... 4 1.8 Scope and Delimitations ...... 4 1.9 Contribution ...... 4 2 Method ...... 5 2.1 Research Approach ...... 5 2.2 Case Study ...... 6 2.3 Critique to Methodology ...... 9 2.4 Report Outline ...... 10 3 Literature and Theory ...... 11 3.1 Industrial Symbiosis ...... 11 3.2 Experiences from Projects for Heat Recovery in Process Industries ...... 13 3.3 District Energy Companies ...... 14 3.4 Data Centers ...... 16 4 Norrenergi ...... 32 4.1 Grid and Production ...... 32 4.2 Market Development ...... 35 4.3 Servitization ...... 35 4.4 Investments ...... 36 4.5 Fuel Mix and Pricing ...... 36 4.6 Production Planning ...... 37 4.7 Risks ...... 37 4.8 Managing District Energy Systems ...... 37 4.9 Data Center Heat Input ...... 38 4.10 Synergies ...... 38 4.11 Obstacles ...... 39 4.12 Heat Recovery Limitations ...... 39 4.13 Development of New Collaborations ...... 39

5 Interview Synthesis ...... 40 5.1 Developments on a Macro Scale ...... 40 5.2 Investment Factors ...... 42 5.3 Availability and Control ...... 44 5.4 Energy and Cooling ...... 45 5.5 Heat Recovery ...... 48 6 Analysis ...... 53 6.1 Synergies for Integrating Data Centers into a Municipal Energy System ...... 53 6.2 Enabling Factors for Energy Recovery from Data Centers ...... 55 6.3 Obstacles to Successful Heat Recovery ...... 58 6.4 SWOT-analyses of Examples of Configuration ...... 62 7 Discussion ...... 67 7.1 Recommended Strategies ...... 67 7.2 Reflections on Results ...... 69 7.3 Reliability and Validity ...... 70 7.4 Suggestions for Future Studies ...... 72 8 Conclusions ...... 73 9 References ...... 74 Appendix A: Interviews with Norrenergi Representatives ...... i Appendix B: Compensation Model for Recovered Heat ...... x Appendix C: Interviews with Data Center Industry Representatives...... xiii

Table of Figures Figure 1 Research process...... 5 Figure 2 Configuration possibilities for connection of data centers to district energy systems. 6 Figure 3 Data center capacity by owner...... 17 Figure 4 Average and maximum rack density...... 23 Figure 5 Hot and cold aisle server rack configuration...... 25 Figure 6 Typical temperatures of components and processes in data centers ...... 28 Figure 7 Potential heat recovery configuration...... 29 Figure 8 Norrenergi district heating grid...... 33 Figure 9 Distributed heat...... 33 Figure 10 Norrenergi district cooling grid...... 34 Figure 11 Preliminary values fuel mix 2016...... 35 Figure 12 Configuration possibilities for connection of data centers to district energy systems...... 62

Table of Tables Table 1 Case analysis structure...... 7 Table 2 Interviewee selection...... 9 Table 3 SWOT-analysis of configuration A...... 63 Table 4 SWOT-analysis of configuration B...... 64 Table 5 SWOT-analysis of configuration C...... 65 Table 6 SWOT-analysis of configuration D...... 66

Acknowledgements This master thesis finalizes our Master’s Programme in Industrial Engineering and Management, with a specialization in Sustainable Power Generation, along with the degree programme in Industrial Engineering and Management (Civilingenjörsprogrammet i Industriell Ekonomi) at KTH Royal Institute of Technology in Stockholm, Sweden. We would like to express our gratitude towards our supervisors Per Lundqvist at KTH and Staffan Stymne at Norrenergi, for their enthusiasm and for encouraging us to pursue the interesting subject of data center integration into district energy systems. We would also like to thank Magnus Swedblom at Norrenergi for valuable input. Finally, we thank our interviewees for sharing their valuable knowledge and for the interesting discussions. Jessica Lind and Erik Rundgren June 2017, Stockholm

1 Background This master thesis explores the possibilities for a district heating and cooling company to integrate data centers for recovery of residual heat. Synergies, enabling factors and obstacles are identified and evaluated in potential collaborations. In this chapter, the district heating industry and the data center industry in Sweden are presented together with a short presentation of existing integration projects between the two industries. In this report, suppliers of district heating and/or district cooling are also referred to as district energy companies.

1.1 The District Heating Market District heating is a well-established industry in Sweden, with the first system built in 1948. The concept of district heating is based on the principle that centralized heat production is more efficient, compared to small local production on the consumption site. Through extensive pipe networks, district heating companies supply space heating and water heating to most Swedish apartment blocks, businesses and public spaces. District heating has been successful in Sweden due to a number of factors: lack of competition from natural gas; strong municipalities when district heating was developed; the “Million Programme”; and climate policy (Magnusson, 2012). The deregulation of the Swedish electricity market in 1996 transformed the district heating market, and district heating has since then been sold at market prices. Historically, district heating has faced competition from oil and electricity. Today, district heating is competing against individual heating solutions for buildings, e.g. heat pumps, in regards to price and efficiency (Magnusson, 2012). Magnusson (Ibid) and Rydén et al. (2013) argue that the local heat pumps, together with increased building energy efficiency and climate change, push the district heating industry into stagnation. Persson and Werner (2011) conducted a study of distribution capital costs in district heating, with regards to different city characteristics in four European countries; Germany, France, Belgium and the Netherlands. They concluded that due to high density in cities, distribution capital costs are low, and district heating will therefore remain competitive in city areas. To diversify the product catalogue, several district heating owners have expanded their systems to include district cooling. District cooling is, compared to district heating, not as extensive in Sweden, and the Stockholm region constitutes more than half of the market. The primary district cooling customers are real estate companies with demand for comfort cooling, and industries in need of process cooling, e.g. data centers (Energimarknadsinspektionen 2013). 1.1.1 Industry Integration Alongside the evolution of combined heat and power plants (CHP), the concept of recovering residual heat has become a way of improving system efficiency as well as decreasing the environmental footprint (Olsson et al., 2015). As part of this development, solutions for recovery of residual heat from energy intensive industries have been introduced. Arnell et al. (2012) defines industrial residual heat as heat in gas or liquid, derived from an industrial process, which cannot be utilized internally as a resource. The heat is dispersed into the surroundings or cooled down (Arnell et. al, 2012). In 2015, residual heat from industries amounted to 7.6 %, approximately 4 TWh, of supplied energy to the district heating networks in Sweden (Svensk Fjärrvärme, 2016). Traditional contributors are process industries, such as

1 steel and pulp. Secondary heat sources make up another contribution from recovered heat. Secondary heat sources have lower temperature, and need to be upgraded in heat pumps before it is fed into the district heating system. Recycled heat derived from secondary heat sources amounted to 5.1 % of the supplied heat in 2015 (Svensk Fjärrvärme, 2016). An increased utilization of industrial residual heat is considered desirable for the energy sector, the industry, and the society as it both optimizes resource use and decreases environmental impact (Ilic and Trygg, 2014). When residual heat and secondary heat sources are used as an alternative to CHP or biomass-fired thermal power, it can decrease both use of primary energy and emissions of carbon dioxide (Arnell et. al, 2012).

1.2 Data Centers Data centers are a growing energy-intensive business and the electricity consumption of the IT-sector is estimated to 7 % of global electricity (Cook et al., 2017). Data centers have a high estimated impact on the Swedish economy. As the industry is in a growth phase, a large part of the positive economic impact comes from construction of new data centers and the business it stimulates. In 2015, data centers were estimated to represent 0.15 % of GDP in Sweden, which is comparable in size to both the textile manufacturing industry and the airline industry. Certain projections estimate data center to constitute 0.45 % of Swedish GDP by 2025. Sweden has approximately 135-150 data centers with a IT-capacity of 0.3 MW and above, foremost located in Stockholm and the counties of Norrbotten, Västra Götaland and Skåne. (BCG, 2016). The Swedish government has introduced a tax reduction on electricity for data centers to attract investments (The Swedish Government, 2016). Data centers have a high demand for cooling and electricity; between 30-40 % of operation spending is estimated to benefit the energy sector (BCG, 2016). An additional challenge is the siting of data centers, as they require high power availability, high cooling capacity and, preferably, proximity to the customers. Proximity to customers means higher security and faster transmission of data. (Greenberg et al., 2009)

1.3 District Heating and Data Center Integration Projects Several district heating grid owners are carrying out projects to integrate data centers for heat recycling (Open District Heating, 2017; EcoDataCenter, 2017). There are also solutions for local recovery of residual heat from data centers to close-by buildings (Fastighetstidningen, 2016). Data centers have been noted by the district heating industry for its potential to be integrated into the grids (Stymne, 2016). Additionally, they strive to be sited close to the customers (Greenberg et al., 2009). The proximity to customers is an important factor for possible integration, as both heat demand in form of a city and an existing district heating grid are prerequisites for heat recovery to district heating.

1.4 Case Company Norrenergi AB, from here on referred to as Norrenergi, is a district heating and cooling company located in Stockholm county with well-developed grids for both district heating and cooling. They have no power production in the facilities, and the business is solely focused on heating and cooling. Their district heating grid covers 90 % of all properties in Solna and Sundbyberg, and parts of Bromma and Danderyd. Their district cooling grid constitutes approximately 20 % of their total grid and provides cooling to offices, hospitals, shopping areas and server rooms (Norrenergi, 2016a). 99 % of the heat input comes from biofuels and

2 heat recycling (Norrenergi, 2016b). Norrenergi owns a heat plant in Solna Strand, Solnaverket, where one of the main sources of heat is water residue of 10 to 18 °C from Bromma wastewater treatment plant. As the Bromma wastewater treatment plant is to be phased out during 2024 (Stymne, 2017), Norrenergi needs to invest in new heat sources, to keep a high fraction of renewable heat production. (Norrenergi, 2016c). To lower the environmental footprint, Norrenergi strives for a high fraction of renewable energy and a cyclic business approach. A cyclic business approach is similar to the concept of circular economy, re-utilizing resources. Heat recovery can reduce primary energy input for district heating and data center cooling combined. Norrenergi is interested in integrating data centers for heat recovery and are evaluating different integration configurations. (Stymne, 2016)

1.5 Problematization District heating is an old industry which will need to adapt to the current changes in the energy sector to increase efficiency and sustainability (Magnusson, 2012). Data centers symbolize a new, fast-growing industry that needs to satisfy a growing demand and secure price competitive cooling options. Defining the parameters for successful integration of data centers into the district heating grid is of interest to both the district heating industry and the data center industry, as it could facilitate long term co-development. Data centers are a growing industry which shows both great economic potential, as well as potential for recovery of residual heat. This offers opportunities for a mutually beneficial relationship between district heating companies and data center operators. There are currently several technical solutions available, and there are examples of data centers connected to the district heating grid in Stockholm (Open District Heating, 2016). Although integration of traditional process industry into district energy systems is well documented in research, data center integration has yet to be studied. Depending on the layout of the integration and the business model used, connecting data centers to district heating grids could mean diversifying the product portfolio of district energy companies. District heating companies, wishing to integrate data centers, will need to consider their market position and value propositions, in order to take full advantage of the new market of heat sources. An adaptability to the demands of data center operation is necessary in terms of temperature levels in different sections of the grids as well as strategic load management. If district heating companies don’t recognize possible mitigation options to counter obstacles, they risk losing market share to local cooling alternatives (Rydén, 2013). Norrenergi is conducting a planning process for future development of Solnaverket and, as a part of this, Norrenergi has reserved on-site space for data centers. To evaluate which configuration is most suitable, Norrenergi needs an assessment of data centers as part of their district heating production, as well as success factors and obstacles. (Stymne, 2016)

1.6 Purpose The purpose of this thesis is to explore the potential for owners of district heating and cooling systems to integrate data centers into their system as a residual heat source. A case study has been conducted at Norrenergi to evaluate the local potential of collaborations.

1.7 Research Question What are the synergies, enabling factors, and obstacles in integration projects between data centers and district heating and cooling companies? How should energy companies account for these factors when formulating product and service offers for heat recovery?

1.8 Scope and Delimitations The scope of this master thesis covers the integration of data centers into the district heating and cooling systems of Norrenergi. It is limited to Norrenergi as company, but some aspects may be generalizable to the district heating industry. Norrenergi has neither electricity production, nor waste incineration, which are thus both excluded from the scope. Even though the work is based on the theory on industrial symbiosis by Chertow (2000), the study does not aspire to be generalizable over different industries. The scope covers four different integration configurations and their characteristics. Geographically, the market analyses are limited to Sweden, but focus on the Stockholm region. The master thesis does not cover any other alternative sources for recovery of residual heat, nor does it include any other possible business development for Norrenergi. Due to limited time and complexity, this study is a single case study, although a multiple case study would have implied a higher validity (Yin, 2014).

1.9 Contribution The research in the field of recovery of residual heat to district heating grids has mainly focused on traditional industries e.g. pulp or steel industry. Examples of this research are Little and Garimella (2011), Arnell et al. (2012) and Björnsdotter (2012). Emerging industries, e.g. data centers, have not been extensively covered in the literature and neither have business solutions for mutual benefit from recycling of residual heat. Pilot projects have been launched by district heating companies, e.g. Open District Heating (Open District Heating, 2016). Albeit numerous studies on third-party access (TPA) (Palm et al., 2009) and pilot projects, and papers investigating technical solutions, no scientific paper covers the business perspective of integration of data centers into district heating and cooling systems. There seems to be a lack of review of the business potential of integration between district heating and cooling owners and data centers. Our thesis will contribute to increasing the understanding of synergies, enabling factors and obstacles in integration between district heating and cooling systems and data centers.

2 Method In this section, we explain and motivate our method to meet the purpose of the study. First, the method is presented, and thereafter the data gathering is explained more explicitly. Lastly, we address critique to our chosen method and model.

2.1 Research Approach The research area in this study includes two different industries with different market conditions and business models. Differences between the industries increases complexity and gives an initially broad scope. We conducted a pre-study with a minor literature review and exploring interviews with Norrenergi, to find a suitable scope. With a defined scope, the literature review was expanded to cover all aspects of the problem. The aim of the literature review was to achieve an overview and understanding of the data center market, an understanding of industrial symbiosis, as well as knowledge about previous heat recovery collaborations with process industries. Based on the knowledge gained from the literature review, we identified both ambiguity within the literature and factors in data center operations, which could impact heat recovery collaborations. The interview questions to both Norrenergi and industry representatives were designed based on the findings in the literature review. Finally, we analyzed how to approach the data center market and the possibilities for integration with Norrenergi. The research process is illustrated in Figure 1. The study is a case study in the sense that Norrenergi and its characteristics represent the energy company perspective. Possible synergies and obstacles related to integration of data centers have been discussed with representatives at Norrenergi, to evaluate their significance and potential. The study evaluates synergies, enabling factors and obstacles to integration and is qualitative in its nature.

Pre-study Problem Literature Interview Analysis formulation review study

Figure 1 Research process. Together with Norrenergi, we singled out four examples for heat recovery configurations when connecting data centers to Norrenergi’s energy system. The configurations have different characteristics and provide different business opportunities. The examples of configuration are visualized in Figure 2. • Type A is a data center located on Norrenergi’s production site. • Type B is a data center connected to both district heating and cooling, and attached to a local heat demand. • Type C is only connected to district heating. • Type D is connected to district heating and has a local heat demand.

Figure 2 Configuration possibilities for connection of data centers to district energy systems.

2.2 Case Study The empirics for this thesis was gathered through a case study. Blomkvist and Hallin (2015) state that a case study can enable the discovery of new dimensions, rich empirical material and is suitable when the purpose is “researching, explaining or describing”. As our purpose is to research and explore the ideal conditions for integration of data centers into district heating and cooling systems, which has not yet been extensively mapped by previous research, a case study was deemed suitable. Cooper and Schindler (2014) describe the case study as an objective to obtain multiple perspectives on a process, situation or event. This is applicable on our thesis, because of its duality of perspectives: district heating and cooling system perspective, and the data center perspective. Applying a case study methodology is sometimes criticized for being unscientific and subjective (Yin, 2014). This can be countered by applying a systematic methodology to the case study (Blomkvist and Hallin, 2015). Yin (2014) stresses that the case study is preferable, when the phenomena studied are not purely historical, but part of a contemporary and, over time, changing process. Interviewing representatives with different perspectives, in combination with written reports, allows for more depth in the study (Cooper and Schindler, 2014). Gathering of data from involved actors of both the district energy perspective and data center perspective decreases the risk of bias towards any party (Yin, 2014). The unit of analysis is Norrenergi, but limited to the aspects relevant to data center integration. The single case study is chosen as method, as the unit of analysis is facing a rather common problem (Yin, 2014): The Norrenergi case is a municipality owned business that is looking to diversify their production through low cost heat input from environmentally favorable sources.

2.2.1 Case Analysis Structure Yin (2014) suggests case studies to be conducted on three levels: system level, intermediate level, and detailed level (shown in Table 1). The system level accounts for trends within the two concerned industries and Norrenergi’s strategy development, with the aim to examine how it can coincide with heat recovery from data centers. The intermediate level discusses synergies and factors related to heat recovery from data center, and aims to provide Norrenergi, as well as other district energy companies, with insights on how to approach and mange collaborations with data centers. The detailed level is related to the configuration possibilities presented in Table 1. The different synergies, enabling factors, and obstacles were related to the configuration possibilities, and a SWOT-analysis (Strengths, Weaknesses, Opportunities and Threats) was conducted for each configuration. SWOT-analysis is a tool for creating insights on how to achieve desired alignment between a commercial venture and its environment (Valentin, 2001). SWOT-analysis is suitable for obtaining support for decision- making processes and assess strategic approaches (Srivastava et al., 2004). It is therefore seen as a suitable tool in the analysis, to give Norrenergi insights, which can be used for further strategic development. The framework of industrial symbiosis, introduced in 3.1 Industrial Symbiosis, gathers all levels of the case analysis structure, proposed by Yin (2014).

Table 1 Case analysis structure.

Norrenergi and Data Center Integration System level Trends and characteristics within the data center and district energy industries. Norrenergi strategy development. Intermediate level Synergies, enabling factors and obstacles in integration projects. Detailed level Obstacle mitigation and recommendations for different configurations of integration.

2.2.2 Sources of Evidence The use of several different sources of evidence increases the validity of the report (Yin, 2014). Data collection has been made through semi-structured interviews and review of published documentation such as whitepapers, trend reports, commercial texts and web pages. A data center industry conference was visited, as is viewed as an additional source of evidence. The findings from the data collection were compared to scientific literature to increase the validity of the conclusions. 2.2.2.1 Literature and Theory To create an understanding of the district heating and cooling business as well as the data center business, we conducted an extensive literature review. The main areas of interest were business models and drivers for the industries respectively, technical specifications for possible integration, and theory on industrial symbiosis. We have reviewed several case studies on the topic of heat recovery from process industries into the district heating system, to encapsulate success factors for profitable heat recovery designs. Parallel to these factors, technical data center specifics have been reviewed, to ensure that there is an overlap in regards to desirable heat flows and temperature levels for both data centers and district heating and cooling systems. Both market analyses and compilation of business models targeted the Swedish markets, as the Swedish district heating and cooling market is well- developed in international standards. The concept of industrial symbiosis is based on the work 7 of M.R. Chertow (2000; 2007) and is introduced to understand the bridging and application of industry integration in general. 2.2.2.2 Interviews The interviews were aimed to cover both district heating and cooling, and the data center industry. All interviewees are listed in Table 2. From Norrenergi, we interviewed Staffan Stymne, head of development and strategy, and Ted Edén, head of distribution. Martin Gierow represents another district energy company, with experience from heat recovery in a research facility with high requirements on availability, similar to data centers. Hans Havtun, lecturer in applied thermodynamics and cooling technology, represents academia, and was interviewed to achieve deeper insight into electronics cooling technologies. To present a correct overview of potential for collaboration, we strived to include perspectives from all segments of the data center industry: • Jonas Dahlgren represents a data center supplier. • Mattias Fridström represents a global connectivity and fiber provider. • Lars Granström represents a colocation and managed hosting business. • Mathias Lindqvist represents a data center supplier. • Jan Lundquist represents a colocation provider. • Pelle Nilsson represents a managed hosting business. • Gert Svensson represents non-commercial HPC. Although no global internet provider is represented, their views are covered by data center suppliers and a global connectivity provider, all experienced in working with internet providers. As Norrenergi and the data center industry represents different perspectives, different interview templates were used.

Table 2 Interviewee selection.

Name Company Position Ted Edén Norrenergi Head of Distribution Staffan Stymne Norrenergi Head of Strategy and Development Jonas Dahlgren Schneider Electric Solution Architect & Product Manager Cooling Mattias Fridström Telia Carrier Vice President and Chief Evangelist Martin Gierow Kraftringen Project Manager Lars Granström Datacenter Chief Technology Officer Hans Havtun Royal Institute of Technology Lecturer in Applied Thermodynamics and Cooling Technology

Mathias Lindqvist Coromatic Energy Expert Jan Lundquist EcoDataCenter Chief Technology Officer, Head Designer Pelle Nilsson 24Solutions Chief Information Officer Gert Svensson KTH PDC Center for HPC Manager

We consider the interviewee selection to be representative for the current data center industry and trends. Blomkvist and Hallin (2015) describe the strive to create a picture from several different perspectives as seeking multiplicity and complexity. During the interviews, we have tried to be impartial, but inquisitive, in our interactions with the informants. For each interview, we reformulated and improved the interview guide to secure it was updated and synchronized with our current knowledge base. The improvement process enabled for complementary questions to be asked at a later time. All answers to complementary questions were added to the interview synthesis. To increase reliability of the study and secure the quality of the data collection, all interviews were recorded and transcribed. After transcription, the data were compiled and divided into a thematic categorization, presented in Appendix C: Interviews with Data Center Industry Representatives. The different perspectives from Norrenergi and the data center industry were compared and analyzed. All interviewees have been given the possibility to review and provide additional thoughts on the information compiled from their respective interview.

2.3 Critique to Methodology A problem with qualitative analyses is the risk of surrendering to biased views of key informants (Yin, 2014). However, we have tried to use multiple sources of evidence to the highest extent. For example, we use documentation together with published articles to complement and affirm the findings from conducted interviews. Documentation refers to event reports, internal records, administrative documents, formal studies, and news clippings (Yin, 2014). This study results in a proposed approach to integration of data centers into Norrenergi’s district heating and cooling systems and is based on a qualitative analysis for different

9 configurations. The quality of the comparison is relying on the lucidity of the qualitative analysis. The integration is discussed as an example of industrial symbiosis (Chertow, 2000), and we have tested the applicability of this theory. Criticism towards industrial symbiosis is discussed in 3 Literature and Theory3.1 Industrial Symbiosis. A critique to this project is that the generalizability might suffer from the limited scope. However, the single case study allows for a more thourough analysis of the actual nature of the problems. Another method may risk losing the detailed perspective in favor of more generalizability.

2.4 Report Outline This section presents the report outline and aims to clarify the report and work structure. Chapter 1 - Background In the background section, we explain the context of the thesis and present our research questions. Chapter 2 - Method The method section explains our work process and provides validity to our report. Chapter 3 – Literature and Theory Literature and Theory is an academic literature review, intertwined with white papers, notes from an industry fair, and other documentation. Chapter 4 – Norrenergi This chapter gives a presentation of Norrenergi based on company information and the interviews conducted with Norrenergi representatives. An interview compilation from the interviews with Norrenergi is found in Appendix A: Interviews with Norrenergi Representatives. Chapter 5 – Interview Synthesis The take-aways from the interviews with data center industry representatives are synthesized. An interview compilation is found in Appendix C: Interviews with Data Center Industry Representatives. Chapter 6 – Analysis In the analysis, we apply the interview synthesis, the Norrenergi perspective, and the literature review on the theories on industrial symbiosis. This chapter contains the main results. Chapter 7 – Discussion The discussion treats recommendations to Norrenergi, reflections on findings and suggestions for future studies. Chapter 8 – Conclusions The conclusions from the study are presented in a short list.

3 Literature and Theory In this chapter, the main areas of literature and applied models deemed relevant to the project are presented. The chapter starts with an introduction of industrial symbiosis, district heating and cooling, and data centers.

3.1 Industrial Symbiosis Industrial symbiosis, named by the power station manager in Kalundborg, Denmark, is subject to several different definitions (Chertow, 2007; Lombardi and Laybourn, 2012; Päivärinne and Lindahl, 2015). Symbiosis is taken from the ecological notion of different species that enhance each other’s living conditions. The definition given by the power station manager reads: “a cooperation between different industries by which the presence of each...increases the viability of the other(s), and by which the demands of society for resource savings and environmental protection are considered” (Chertow, 2000). Chertow et al. (2008) propose three different opportunities for resource exchange in a symbiotic industrial relationship: 1. By-product reuse. 2. Utility/infrastructure sharing. 3. Joint provision of services.

In practice, industrial symbiosis is often studied as a product of eco-industrial parks (Chertow, 2000). The eco-industrial park (EIP) is a more limited notion, that also implies geographical proximity as part of the relationship between the parties. Chertow (2000) states that the symbiosis doesn’t have to occur within the strict boundaries of a park, and points out that the EIP in Kalundborg started out as partnerships between nearby, but not co-located businesses. However, the proximity requirement has come to be questioned, for cases where it is possible to transport the resources without deteriorating of the inherent value (Lombardi et al., 2012). The EIP is a cluster of unrelated industries that engage in exchanges, just like industrial symbiosis, but Lombardi et al. (2012) argue that it usually includes a greater role in society. In relation to EIP, Lombardi and Laybourn (2012) do not acknowledge the consideration of the “green” aspects of the initiatives as a requirement for industrial symbiosis, as long as the benefits can be considered “green”. They draw from experience that initiatives are usually conducted out of self-interest, rather than from an urge to foster an environmental society. Instead, Lombardi and Laybourn (2012) introduce a new definition of industrial symbiosis: “Industrial symbiosis engages diverse organizations in a network to foster eco-innovation and long-term culture change. Creating and sharing knowledge through the network yields mutually profitable transactions for novel sourcing of required inputs, value-added destinations for non-product outputs, and improved business and technical processes.” This definition draws specifically on the mutual benefits of the parties, and gives less consideration to the ecological aspects. Naturally, as businesses become more resource efficient, the society reaps environmental benefits. Chertow (2007) identifies two main models of industrial symbiosis: planned EIP model and self-organizing symbiosis model. The planned EIP is usually organized by a third party, such as a government agency, and is the product of a conscious effort to identify and attract different industries and co-locate to create symbiosis. The self-organizing model is purely based on private decisions on cost reductions, revenue enhancement and business expansion,

11 and does usually not rely on conscious acts toward symbiosis. Reichenberg et al. (2011) indicates that most examples of industrial symbiosis are self-organized EIPs, and that it might be hard for third parties to plan and create successful EIPs. However, as an EIP may start out as a self-organizing entity, it can in time attract attention from both governments and other investors. Chertow (2007) also argues that a major actor, e.g. power station or large industry, can function as an “anchor tenant”, attracting businesses to form an EIP. Behera et al. (2012) discuss methods to facilitate successful development of industrial symbiosis, e.g. partners should be chosen based on geographical proximity and willingness to collaborate in synergies. Additionally, Behera et al. (2012) state that industrial symbiosis must be achievable by current available technologies, benefits should exceed the investments, and that the proposed industrial symbiosis must be legal. Chertow (2000) presents two approaches for attracting new businesses for integration. The business-based approach means that an existing business is trying to find partners among the existing tenants, while the stream-based approach means that industrial symbiosis is created through identification of resource streams for a whole system. As the stream-based approach needs a planning function from an external organization, this approach becomes more complex. To mitigate the complexity, Chertow (2000) suggests a mixed approach, where new tenants are proposed to invest in an already developed area based on their respective streams.

3.1.1 Energy Companies as Anchor Tenants Data centers and district heating and cooling systems are two unrelated industries that, when clustered together, could offer opportunities for resource efficiency in terms of the conditions for industrial symbiosis set by Chertow (2000). Reichenberg et al. (2011) argues that a municipality owned energy company, given its position in society, could function as an anchor tenant and thereby attract businesses to the region.

3.1.2 Applying a Business Perspective Päivärinne and Lindahl (2015) state that industrial symbiosis has been criticized for a lack of consideration for the business perspective, and only focusing on the inputs and outputs of collaboration. Instead, they point out the importance of considering the involved actors’ business models. Tsvetkova and Gustafsson (2012) state that business integration was a challenge for solutions based on industrial ecosystems thinking, and that environmental and economic benefits to be gained in more complex networks, e.g. in industrial symbiosis, are often overlooked. Boundary-spanning business models are mentioned as a mitigating strategy, requiring deeper stakeholder involvement from customers, suppliers and partners. Additionally, modularity is discussed as a strategy for enabling mass-customization in industrial ecosystems. As local conditions for industrial collaboration differ between locations and stakeholders, modularity in solutions offerings allows for flexibility and possibilities to adapt to local conditions, while still achieving a certain level of standardization. The authors also emphasize the customer as an important element of the business model, as the same product may hold different values for the customers on different markets. (Tsvetkova and Gustafsson, 2012) To add a business model perspective to industrial symbiosis, Päivärinne and Lindahl (2015) propose a combination of industrial symbiosis with the concept of Integrated Product and Services offering (IPSO). The concept of IPSO aims to reduce the consumption of products

12 by alternative scenarios of product use, and simultaneously increase a company’s competitiveness and profitability. Applying the IPSO concept of recovery of residual heat would entail viewing solutions for heat recovery as services rather than products. Päivärinne and Lindahl (2015) explore the implications of a combination of industrial symbiosis and IPSO to facilitate increased utilization of excess heat. They attempted to answer the question of why there is so much unutilized excess heat, by conducting qualitative interviews with actors in five different cases of unutilized excess heat. The study established that although potential benefits for collaboration exist, inter-organizational collaborations are often difficult to initiate and that it is important to have common goals and transparency behind a business agreement. According to Päivärinne and Lindahl (2015), previous studies have also concluded that the main difficulties with collaborations between energy companies and industrial firms are caused by lack of knowledge and understanding of each other’s businesses. Additionally, an industry can be reluctant to enter an agreement for heat collaboration as heat is not a part of the industry’s core business, mainly connected to a fear of disruption of the industry’s main activities. (Päivärinne and Lindahl, 2015) Päivärinne and Lindahl (2015) state that an increased use of excess heat could be enabled by an IPSO provider with competence and responsibility for distribution of heat. The offerings would facilitate heat transaction for both supplier and recipient and an IPSO could be formulated in different ways, depending on whom it is addressed to. Päivärinne and Lindahl (2015) conclude that the service offerings should be formulated in such a way that the risks are reflected in the potential profits, and that a combination of industrial symbiosis and IPSO should be applied for collaborations for heat recovery, to address both organizational and business factors.

3.2 Experiences from Projects for Heat Recovery in Process Industries Recovery of residual heat from process industries is well established, and can be considered examples of industrial symbiosis as a by-product from one actor is reused by another. There are several examples documented in previous research through case studies. Arnell et al. (2012) studied cases on the Swedish west coast, in Oskarshamn and in Oxelösund. Björnsdotter (2012) studied the recovery of residual heat at SSAB EMEA in Borlänge, and McEwen (2015) studied the potential for heat recovery from industries in Västerås. The collaboration is usually organized between a local utility company and a nearby industry with heat as a by-product of their operations. The majority of delivered residual heat stems from the pulp and paper industry, the petrochemical industry, the iron and steel industry, and the mineral oil refineries (Arnell et al., 2012). Previous case studies state that realization of recovery of residual heat from industries is dependent on the project being economically viable for both parties. For many district heating companies, cogeneration plants for heating and electricity are often more profitable. This is partly due to policy instruments, such as the electricity certificates, favoring production of renewable energy, as well as the fact that cogeneration allows for electricity production with a high degree of efficiency (Arnell et al, 2012; Björnsdotter, 2012). There are many approaches to pricing of residual heat, depending on the nature of the collaboration. In some cases, the pricing is related to how the necessary investments have been divided between actors (Björnsdotter, 2012). Difficulties in establishing collaborations for residual heat can arise from the different business models of the energy company and the

13 industries, as they may have different operation goals (McEwen, 2015). For example, there are differences between a profit-maximizing industry and a municipally owned energy company. Difficulties may also arise from different views of heat as a product, where it for the energy company is part of the core business, but for the other industry is simply a by- product. Therefore, it can be more difficult for the industry to motivate investment for heat recovery, compared to the energy company. Another difficulty that may arise is if there is not enough heating demand connected to the grid to meet the supply from the connected industry (Björnsdotter, 2012). Besides economic viability, previous case studies noted pricing and reliability of heat supply as important factors for contract negotiations. (Arnell et al, 2012; Björnsdotter, 2012). Energy companies have expressed concern for becoming dependent on heat from industries, which would be problematic if heat deliveries would default. The energy company has a commitment to their consumers and risk facing reprisals if the supply temperature cannot be guaranteed (McEwen, 2015). Other important factors for negotiations were contractual periods, risk management, political decisions, and engagement (Arnell et al, 2012). There are examples of previous cases, where key individuals in the energy company had previous work experiences from the industry in question, which benefited the collaboration (Arnell et al, 2012). Participants in collaborations for recovery of residual heat have also mentioned that it is to their advantage to have key individuals within the companies to drive the project forward (Arnell et al, 2012; McEwen, 2015). Representatives from industries have also mentioned that it was beneficial to have someone within the company with expressed responsibility for the energy issue and energy efficiency. (Arnell et al., 2012). This is in accordance with Rydén et al. (2013), who conclude that one success factor for the competitiveness of district heating companies’ business model are designated project managers, which strongly contribute to the success of the company. In their research, they found that several energy companies had in fact recruited people with competence in forest industry to fully take advantage of biofuel possibilities. In addition to this, open communication and trust have also been identified as success factors. (Arnell et al, 2012; McEwen, 2015).

3.3 District Energy Companies District heating is an established industry which in later years has further developed with the addition of district cooling. District energy companies is used to refer to companies which provide both district heating and district cooling. The following chapter discuss trends on the district heating market and current business model development for district energy companies.

3.3.1 Competitive Business Models for District Heating and Cooling The district heating operators in Sweden are facing several major challenges as the market conditions have changed in later years (Rydén et al, 2013). Among the challenges are a maturing market, tougher competition from local heat pumps, a warmer climate and better insulation in buildings. Commenting an extensive market analysis, Rydén et al. (2013) stress the importance of a well updated business model to manage in the new environment. Instead of focusing on volume growth, the district heating companies need to work with pricing, communication and taking advantage of existing resources. Kindström et al. (2015) discuss the increased need for energy companies to focus on more complex value propositions to customers, rather than just product delivery of heat and electricity. More holistic customer 14 solutions should offer more customer value and increased possibility of adapting to specific customers. This would enable advantages such as deeper relationships to customers and new possibilities for differentiation, growth and increased profitability (Kindström et al., 2015). Brecheisen and From (2013) noted a dissatisfaction among data centers who were customers of district cooling, as they found the pricing to be outdated. Especially cooling customers felt that they had to pay for a service, that was easily accessible by local, cheaper, alternatives (Brecheisen and From, 2013). An increased focus on providing customer services, servitization, often requires not only a change in customer offer, but also internally in the company and its business model. When increasing focus on servitization, Kindström et al. (2015) state that it is important to have a clear strategic agenda to achieve a good overview and alignment between different initiatives. Dedicated roles for the initiatives will help upscaling from a few pilot customers to a larger part of the customer base (Kindström et al., 2015). Selling services can for some energy companies create need for new competence internally and there are different strategies to achieve this new competence as well as an efficient use of resources. In the study by Kindström et al. (2015), several companies used teams or networks of subcontractors to achieve a holistic competence. Another, less common, strategy was to attempt to standardize service through modularization. Changing from sales of products to sales of services can also prove to be a challenge as it can require a different and more complex sales process. An increased focus on energy services can however increase the value of the brand from both a sustainability perspective, as well as a competence perspective (Kindström et al., 2015). Reichenberg et al. (2011) declare that the main driver for energy cooperation is that the cooperation is mutually beneficial for the parties. The roles of a partnership need to be settled. This means that customers delivering an added value need to be recognized for their role in the business and not carry the costs alone. Rydén et al. (2013) conclude that the customers need to be more involved in the business and that the goal is a partnership relation.

3.3.2 Customer Segmentation The study conducted by Kindström et al. (2015) discovered that several energy companies did not prioritize segmentation of customers. Segmentation of customers is key, especially when delivering energy services as they can be more differentiated and heterogenous compared to traditional energy products. Successful energy services that deliver value to the customer require both the energy company to have a deep understanding for the customer, but also the customer to understand the value of the provided services. The energy company has to be able to properly communicate the value of the service to the customers. These more complex processes pose a risk of increased transaction costs but which can be managed through differentiated pricing and pricing strategies. (Kindström et al., 2015)

3.3.3 The Price Dialogue The Price Dialogue is a model developed by Riksbyggen, the Swedish Association of Public Housing Companies (SABO) and Swedenergy AB to strengthen the customer’s position and contribute to an increased trust to the pricing of the district heating suppliers. This is achieved by a yearly price meeting where the local pricing model is reviewed together with the next year’s pricing. The pricing for the coming years are forecasted. This dialogue is conducted

15 between the district heating supplier and its customers. After the pricing model has been discussed between customers and supplier, it is reviewed by The Price Dialogue secretariat. District heating companies applies for membership in The Price Dialogue and the membership is dependent on a correctly conducted yearly review of the pricing model. (The Price Dialogue, 2017)

3.3.5 Security of Heat Supply District heating is a critical societal function, with large-scale infrastructure. A disruption in distribution or production risk affecting many consumers. However, there are no constitutional requirements regarding availability of district heating and there is no specific authority responsible for control of district cooling. The district heating industry has determined distribution to be the most critical link in the heat supply chain. Underground pipes are exposed to stress and leakages can be difficult to discover. Modern distribution pipes are of better quality compared to those built during the expansion of district heating. Modern pipes experience less disruptions, but old installations are still present within the systems. Pipes are more exposed to changes in cities compared to rural areas, due to urban development. (The Swedish Energy Agency, 2016) Level of redundancy and readiness differ between different energy companies, but there is in general a large dependency on electricity production. Electricity is required to operate pumps, fans, boilers and control systems. Usually, there is not enough reserve power to maintain heat production and distribution during a power outage. Incentives to increase redundancy and create the ability to operate the system without electricity is limited, as end consumers usually cannot receive heat during disruptions in power supply. District heating companies are well prepared to manage technically related damages and incidents, but less prepared for disruptions due to external infrastructure or organizations on which they are dependent. Nor are they properly prepared for risks related to information security. There are ongoing efforts for developments in this field within the district heating industry. However, the process until all companies perform proper risk and vulnerability analyzes is deemed to be long. In regards to district heating companies’ financial positions, most are well prepared to manage a changing environment with a good long-term solvency. Currently, due to differing levels of readiness between district heating companies, impact of disruptions for the end customers vary depending on supplier. (The Swedish Energy Agency, 2016)

3.4 Data Centers The following chapter explores the data center industry and its characteristics. Data center actors, market trends, data center cooling and heat recovery are discussed. The chapter is meant to give a comprehensive understanding of the data center industry and the characteristics which will impact potential for heat recovery collaborations.

3.4.1 Data Center Actors The owners or operators of data centers in Sweden can be roughly divided into three categories; enterprises, third-party providers and global internet companies. Enterprises are large companies which own and operate their own data centers to support main operations, and the capacity of the in-house data centers are often below 0.3 MW. Most enterprises will prefer to keep their data centers close to corporate headquarters to facilitate management and maintenance. Most third-party providers operate data center around 1 MW, while global

Internet companies foremost operate hyper-scale data centers above 10 MW, see Figure 3. (BCG, 2016)

Figure 3 Data center capacity by owner. (BCG, 2016) For third-party providers, data centers are core assets as their main business constitutes of renting rack and server space. Third-party providers can roughly be divided into colocation, managed hosting and Data Center as a Service (DCaaS) (BCG, 2016). In a colocation solution, the customer of the third-party provider purchases the servers and other necessary hardware, as well as necessary software. The customer is responsible for the configuration, and then installs the equipment in the colocation provider’s data center. The colocation provider provides the necessary infrastructure and is responsible for security and maintenance of the data center. Managed hosting is instead where the third-party provider is responsible for all necessary hardware and software for the customer. Managed hosting can entail both sharing severs with other customers, or having a dedicated private server (Aboutcolocation, 2017). DCaaS offers an offsite data center infrastructure and, in comparison with colocation, increases the possibility to customize the facility to the customer’s needs in terms of volume and capacity (Serverlift, 2012). Global internet companies offer cloud services, which entails remotely processing and storing customer data. The customers are not tied to any specific server, but merely pay for accessing the computing capacity and desired applications, which entails that the data centers are often location agnostic. This means that in regards to location, they are foremost concerned with being located on the same continent as the intended customers. (BCG, 2016). There are several variations of cloud services, with varying degrees of privacy and encryption (Coromatic, 2015). There are also smaller cloud-computing companies, which may rent space at a colocation provider. (BCG, 2016) A trend within enterprise IT management is hybrid IT, which combines in-house servers with cloud services. The perceived advantage of hybrid IT is the combination of the scalability and flexibility of cloud services, with the security and control of in-house premises. (HP, 2017)

3.4.2 Market Trends The development of the data center industry is influenced by a number of market trends. The main trends discussed in the following segment are based on information gathered at a data center industry fair in Stockholm 2017, as well as market reports. 3.4.2.1 Development of Actor Segments The consultancy firm Boston Consulting Group (BCG) consider global internet companies and location-independent third-party providers to represent much of future growth in the data center market. They assess that the trend is moving towards consolidation and outsourcing. A macro analysis by the consultancy firm Trendie (2017) also states that there is an ongoing outsourcing trend and that the segment of third-party providers is expected to grow. Some enterprises will phase out in-house data centers, but keep ownership of the severs and move them to colocation providers. Other enterprises will move directly towards cloud services. Both behaviors would entail a decrease in location dependence (BCG, 2016). IDG Connect Nordics conducted a quantitative study (2016) on data center market trends for Digiplex. They interviewed individuals from companies in different industries, in three leading positions: CEO, Chief Financial Officer (CFO) or Chief Information Officer (CIO)/Chief Technology Officer (CTO), concerning their companies view on IT services. 55 % of the responding companies currently operate in-house data center, but the study showed this number would drop to at least 43 % within two years. This study was thus in accordance with BCG assessment of a trend towards outsourcing. The study showed the main challenge of respondents’ current data center solution to be; staffing, that it takes away focus from core business, and difficulties keeping up with increased data quantities. The dominating perceived advantage of an outsourced solution was scalability and flexibility and that the data center would then be managed by professionals and the possibility of a tailored solution. (IDG Connect, 2016) At the DI Datacenter conference in Stockholm 2017, the focus on the discussions were how companies in Sweden could distinguish the Swedish market to attract the data center operation of large, global internet companies. Representatives from several actors, such as Interxion, Huawei and Coromatic stated that so-called hyper-scale data centers are increasing (Bank, 2017; Bäckström, 2017; Lindqvist and Hedman, 2017), and Digiplex and Huawei said that there is an ongoing consolidation globally in regards to hyper-scale data centers (Eckhoff, 2017; Bäckström, 2017). Peder Bank (2017), managing director for Interxion in Sweden, said he believes that hyper-scale data centers will grow as well as the smaller, interconnected data centers closest to the customers. He estimated that the share of colocation actors will remain approximately the same, and that in-house data centers will decrease in numbers. Many data centers today, especially hyper-scale data centers, are constructed in rural areas. Third-party provider are expected to see the benefits of locating the most data-heavy traffic close to population centers, while the bulk of traffic is moved to locations with other advantages such as stable, renewable energy at low prices (BCG, 2016). According to a review conducted at the Data Center World conference in 2014, 32 % of respondents stated operating costs to be the main factor influencing data center location, while 29 % stated proximity to operations and headquarter staff (Mortenson, 2014).

3.4.2.2 Time to Market In order to attract the large, global internet companies with hyper-scale data centers, time to market has to be short (Lindqvist and Hedman, 2017; Sokolnicki, 2017; Svanberg, 2017; Egnell, 2017). Once an actor has decided upon a location, everything should be in place to initiate construction as quickly as possible. Building permits, architectural plans etc. should all be in place. Business Sweden have teams that work with sales and preparation of sites in order to attract investments. A site has to be “shovel ready” once a customer decides for an establishment (Sokolnicki, 2017). Trendie (2017) also emphasizes the importance of time to market, but explains that the process of establishment differs depending on type of data center. Large data centers above 5 MW have established standards and partners for construction, while smaller data center companies are more dependent on other industry actors. (Trendie, 2017) 3.4.2.3 Availability and Control Data centers have high requirements for availability to ensure user trust in the provided services. Level of availability is a trade-off between the cost of failure and the cost of preventing them (Barroso et al., 2013). A report by Emerson Network Power (2011) stated downtime could impact the profitability of some data centers, and that relationship between data center availability and Total Cost of Ownership (TCO) has been enforced. The majority of root causes for downtime was attributed to vulnerabilities in data center power and cooling infrastructure. The most significant cost consequences were not directly related to the infrastructure or detection of error, but instead related to business disruption. There are many strategies for minimizing risk of downtime: applying refrigerant-based cooling instead of water-based cooling to minimize risk of water incursion; eliminate hot spots by applying row- based precision cooling; proper monitoring systems; preventive maintenance. Mitigating for risk of downtime increases initial investment. (Emerson, 2011) A high level of availability (uptime) is essential, especially for third-party providers (Trendie, 2017). Vyncke (2017) stated that there has been an increase in risk awareness amongst their customers at Rockan Datacenter. Egnell (2017) also stated that data center suppliers have experienced an increase in security demands. 3.4.2.4 Energy Efficiency and Heat Recovery The opinions at the industry fair regarding the potential and the importance of recycling heat from data centers were diverse. Graf, CEO of Hydro 66, declared that there is no reason for them to recycle heat from their data center located in Boden, as it is already very energy efficient (Graf, 2017). Bank (2017), however declared that one of the reasons for why Sweden is attractive for establishment of data centers, is the possibility of recycling heat. Representatives from the data center supplier Coromatic toned down the importance of free cooling, implying that this did not provide any particular edge, and that recycling residual heat was of greater importance. They also stated that the reduced energy tax for data center (explained in section 3.4.3.1 Energy Tax for Data Centers) reduced the economic incentive for improving energy efficiency and recycling heat. (Lindqvist and Hedman, 2017) The survey conducted at the Data Center World conference concluded that as the main area of improvement in their operations, 19 % of data center operators stated energy efficiency and 15 % stated a better cooling system. 84 % also responded that they probably or definitely feel a need to consider renewable energy to manage future data needs. (Mortenson, 2014) A

19 review by Data Center Dynamics concluded that the amount of data center actors which specified an environmentally friendly and more sustainable profile as a reason for investment increased from 30.5 % in 2014 to 40.2 % in 2015 (DCD Intelligence, 2015). Data center requirements for safety of operations are a challenge for district heating and cooling solutions. Especially foreign companies are seldom familiar with the concept, and for them it thus entails additional uncertainty and risk. (Trendie, 2017) 3.4.2.5 Modularity and Scalability A current trend in data center construction is modularity and scalability, especially within the colocation segment. Colocation providers, compared with other segments, invest more in modular facilities (DCD Intelligence, 2015). Flexibility is important and construction should be modular and scalable in order to grow with customer demand (Egnell, 2017). Modular construction enable higher utilization of facilities as it can otherwise take years to fill up installed capacity (Bäckström, 2017). For data center customers, turning to a colocation provider shift capital expenditure to operational expenditure. Instead of investing in an in- house facility, the customer can rent capacity and only pay for actual usage. This enables the customer to successively grow IT-service as demand grows. (Industry Perspectives, 2015) According to Ebrahimi et al. (2013), data centers rarely utilize more than 20 % of maximum server capacity, but both Greenberg et al. (2009) and Davies et al. (2015) estimate utilization level to 10 %. Reasons for this could be uneven applications fit, uncertainty in demand forecasts and risk management (Greenberg et al., 2009). However, recent improvements such as consolidation and virtualization of servers achieve higher IT workloads, up to 50 % utilization (Davies et al., 2015). 3.4.2.6 Capital Expenditures vs. Operational Expenditures Eckhoff, CEO at Digiplex, states that financing is very important to data centers and that Digiplex’s advantage is its financially strong American owners. (Eckhoff, 2017). Head of advisory at Coromatic, a supplier to data centers, highlighted the possibility of moving capital expenses to operational expenses by offering Data Center as a Service. By applying flexible pricing and servitization, it is possible to build and scale according to the customers’ needs. (Egnell, 2017). The customers can then avoid making large initial investments and thus increase working capital (Coromatic, 2017). 3.4.2.7 Edge Computing Increased distributed data from several applications strains the network capacity. This is partly due to the growth of for example the Internet of Things (IoT), virtual reality (VR) and smart cars. To reduce strain on network capacity and achieve low latency, some data computing will move to local devices and sensors together with regional data centers, instead of the large, global data centers. (Miller, 2017). This is called edge computing and can be compared with interconnected data centers close to the customer. They will be essential for companies who need to provide their customers with low latency, such as services for video streaming. (Miller, 2015).

3.4.3 Attractiveness Factors for Investments in Sweden At the DI industry fair (2017), renewable and environmentally friendly electricity at low cost was listed as one of the main advantages of locating data centers in Sweden and the Nordics, compared to other companies globally (Lindström, 2017; Sokolnicki, 2017; Svanberg, 2017;

Bank, 2017). In the Data Center Risk Index provided by Cushman & Wakefield (2016), Sweden places as the fifth most attractive location for establishment of data centers, with respect to the top risks likely to affect data center operations. Sweden places well in regards to ease of doing business, economic and political stability, share of renewables in energy supply, energy security, water availability and international bandwidth. It does not place as well in regards to energy/electricity cost per kWh, corporation tax (though not yet accounting for the reduced energy tax, see 3.4.3.1 Energy Tax for Data Centers) and GDP per capita. The four top countries placing above Sweden are all European and three out of four are part of the Nordics. They place in the following order (1-4): Iceland, Norway, Switzerland and Finland. (Cushman & Wakefield, 2016) 3.4.3.1 Energy Tax for Data Centers The Swedish Government imposed a reduced electricity tax on data centers, starting the 1st of January 2017. The reduced tax applies to both existing and new data centers with a capacity above 0.5 MW. (Johansson, 2016). The tax on consumed electric power is reduced to the same level that applies to the manufacturing process in industrial operations, 0.05 SEK/kWh. This is intended to increase the competitiveness of the Swedish data center industry and improve the conditions for new establishments. (The Swedish Government, 2016). This decreases operating costs significantly for data center as it represents a 97 % tax cut. As an example, a 10 MW data center can save between 30-50 % in costs. (Node Pole, 2016) The government budget proposal for 2018 contains a proposal to expand the tax reduction to include smaller data centers with a capacity of at least 0.1 MW (The Swedish Government, 2017). The tax is current only applicable for electricity consumed directly by the data center, and does not include electricity consumed for producing heating or cooling to data centers. A new referral to the Council on Legislation has been submitted by the government, proposing a change in to include recipients who have consumed electric power for a purpose which supports the data center operations. (Andersson and Bergqvist, 2016)

3.4.7 Data Center Investment Drivers Traditionally, data centers have been focusing on securing uptime and availability to the customers. The dependability of the technical equipment in data centers is now considered high enough to shift focus to expenses in the long run. Today, the Swedish market for data center investments is price focused, with operational costs as main driver (BCG, 2016). Although environmental performance is considered a growing interest, it has not yet grown to affect investment decisions in large scale (Brecheisen and From, 2013). However, new “green” partnerships between large-scale data centers and district heating operators are launched in several municipalities across Sweden (EcoDataCenter, 2017; Stockholm Data Parks, 2017). Colocation actors have made a shift in focus towards IT and cloud services, in order to stay competitive. They have increased investments in facility equipment, IT-solutions, and optimization. The main reason for investment has been determined as need to increase IT capacity while simultaneously reducing operating costs. Actors see a need to apply technologies such as virtualization and the cloud in order to meet a changing customer demand. (DCD Intelligence, 2015)

3.4.8 Cost Analysis Capital expenditures (CapEx), such as construction costs, are paid upfront but depreciated over a longer time frame. The CapEx vary greatly depending on design, size, location and speed of construction. Degree of redundancy and availability also affect capital expenses. The monthly amortization cost of construction costs depends on the expected lifetime of the data center and assumed interest rate. (Barroso et al., 2013) The lifetime of a data center is commonly assumed to be between 10-15 years (Barroso et al., 2013; Rasmussen, 2012). Servers are assumed to have a shorter lifespan and are depreciated over 3-5 years (Barroso et al., 2013; Greenberg et al., 2009). Operational expenses for data center can vary depending on several characteristics: operational standards; design; age; size; and geographic location. One of the major cost drivers for operational expenses is power consumption. Capital budgeting is a method for evaluating a project investments by evaluating future cash flows and its impact on a company’s available cash, either by calculating free cash flow of net present value (NPV) (Berk and DeMarzo, 2014). Another method for evaluating investments is The Total Cost of Ownership (TCO). TCO of a data center includes both capital and operational expenses and is a tool for business decision processes or calculating return on investment. There are no standards for measuring TCO and several metric systems are in use such as cost per square meter, per server, per watt. Barroso et al. (2013) claims cost per area to be difficult to apply when comparing projects, and cost per critical watt (W) to be a more useful metric. Cost per critical watt, or cost per IT-equipment power illustrates cost per watt utilized by IT equipment, and is thus related to one of the primary cost drivers. A data center built with a high degree of redundancy will be more expensive per critical watt, compared to a data center with lower degree of redundancy, even though it is otherwise identical. Rasmussen (2011) suggests in a white paper that TCO should account for utilization of racks, as this would relate costs more directly to the installed IT-equipment. Barroso et al. (2013) modelled two cases of multi-megawatt data center in the United States. Case A included more expensive servers with a peak power draw of 340 W per server, and the Case B included cheaper servers with peak power draw of 500 W per server. Cost of capital was set at 8 %. In case A, server amortization was 65.9 % of total TCO, and in case B 29.5 %. However, server power cost in case B represented 14. 3 % in case B, compared to 3.8 % in case A. Total server costs, that is depreciation costs and associated maintenance costs, proved to be a major cost driver of data center TCO in both cases, followed by capital expenses for the data center infrastructure. The data center infrastructure is the facilities dedicated to power delivery and cooling (Barroso et al, 2013). Greenberg et al. (2009) modelled amortization costs for a data center consisting of 50 000 servers and cost of capital at 5 %. This study concluded that server costs constitute roughly 45 % of total data center costs, followed by infrastructure costs at 25 % and estimated costs for power draw at approximately 15 %. The total power cost is dependent on the electricity price in the market where the data center is located, and the amount of power utilized by the data center. The cooling system and the IT- hardware (including the servers) constitute major parts of power consumption (Rasmussen, 2011b).

3.4.9 Data Center Cooling A data center can be described as a space dedicated to Information and Communications Technology (ICT) infrastructure and hardware. Servers are placed in server racks and both server thickness and rack height are measured in the unit “U”. Server thickness is usually 22 between 1-2 U, and a standard full size rack is 42 U (Ebrahimi et al., 2013). Server operation requires infrastructure to secure power supply, fiber connectivity, and cooling (Greenberg et al., 2009). The applied cooling technology in a data center affects the temperature difference of the cooling medium and its return temperature, an important factor affecting efficiency of heat recovery. Larger temperature differences improve the efficiency of heat recovery and cooling. (Ebrahimi et al., 2013). 3.4.9.1 Temperatures The dominating temperature guidlines for data centers are produced by ASHRAE, American Society of Heating, Refrigerating and Air-Conditioning Engineers. They continuously publish a handbook with guidelines for ICT environments, applied by the data center industry. Data center environments are continuously developed to save energy and reduce operational expenses and many data centers currently operate in several degrees warmer compared to 10- 15 years ago. New technologies for energy savings enable increased temperatures. Current ASHRAE thermal guidelines range from 5 oC to 45 oC, depending on operating class. (ASHRAE, 2016) According to data which was self-reported to DCD Intelligence however, the global average of inlet air temperature is 22.5 oC. Only 2 % of respondents reported an inlet temperature of 27 oC or warmer, and 9.8 % reported 18 oC or cooler. (DCD Intelligence b, 2015) 3.4.9.2 Power Density According to a world-wide survey conducted in 2016, average power density was 3.8 kW/rack during 2015. Europe had a slightly higher number at 4.3 kW/rack. Average maximum density was 9.6 kW/rack globally and 13.2 kW/rack in Europe. The organization type with both highest average power density and highest average maximum density was IT services. With 6.3 kW/rack and 14.6 kW per rack respectively. Colocation and telecom actors had an average density of 5.6 kW/rack and an average maximum density of 10.7 kW/rack. The survey also indicated a slight future increase in power density, see Figure 4. (DCD Intelligence, 2016)

Figure 4 Average and maximum rack density (kW/rack). (DCD Intelligence, 2016) According to Ebrahimi et al. (2013), there is a direct proportionality between data center costs and floor area, which incentivize manufacturers to continuously produce modules with higher density.

3.4.9.3 Air-Cooled Systems The most common data center cooling designs use either a Computer Room Air Conditioning (CRAC), or a Computer Room Air Handling (CRAH) unit to supply cold air (DCD Intelligence b., 2015; Davies et al., 2015). When applying a chilled water system, larger chillers produce cold water to cool the supply air, which then circulates in the data center. Unless the residual heat in the air is recycled, it is usually cooled down by utilizing a cooling tower (DCD Intelligence b., 2015). The term CRAC unit can also be applied when using a chilled water loop (Barroso et al., 2013). The server racks are commonly arranged in hot and cold aisles: the cold aisles are facing the front of the servers, and the hot aisles are facing the back of the servers (Ebrahimi et al., 2013). The cold supply air flows through the servers and then exits in the hot aisles. The hot air is thereafter circulated back for cooling (Ebrahimi et al., 2013). Efficient air cooled systems can give a temperature different between supplied and returned air of 15 oC, and according to Ebrahimi et al. (2013), cold air is typically supplied at 10-32 oC and the exhaust air is returned to the CRAC unit at 50-60 oC. The most common design for distributing the air is either through a raised floor, but air distribution through the ceiling is also relatively common (El Azzi and Izadi, 2012). An example of a hot and cold aisle server rack configuration where air is supplied through the floor can be seen in Figure 5. The major cooling issue in hot and cold aisles approaches was, according to Ebrahimi et al. (2013), turbulent mixing of hot and cold air. El Azzi and Izadi (2012) also discussed mixing of hot and cold air flows as a significant inefficiency in data center cooling technologies, and additionally they discussed non-uniform heat loads as a reason for inefficiencies. Due to a mix of heterogenous hardware in the server racks, the heat load in the racks are not uniform. This entails non-uniform cooling loads for the CRAC-units and can cause both under- and over provisioning, where the CRAC units either supply air which is too hot, or they supply too much cooling as they are designed for an increased cooling load. One proposed solution to this problem was to design air conditioners with variable capacities, to enable the proper cooling load for a specific heat load (El Azzi and Izadi, 2012). Other alternative methods for increasing energy efficiency of air cooling is hot and cold aisle containment, where one of the aisles are contained to prevent mixing of the different air streams (Davies et al., 2015).

Figure 5 Hot and cold aisle server rack configuration. (42U.com, 2009)

3.4.9.4 Free Cooling Free cooling is a method that utilizes low temperature, local ambient conditions and does not require a refrigeration cycle. However, free cooling still require power to operate pumps and fans (DCD Intelligence, 2015). As previously mentioned in 3.4.3 Attractiveness Factors for Investments in Sweden, the opportunities for free cooling makes a strong case for locating data centers in Sweden and the Nordics, and especially in the northern parts. There are two main methodologies for free cooling; air-side systems and water-side systems. Air-side systems bring outside air into the data centers by passing it through filters. Filters are applied to avoid particles and other contamination of the IT equipment, as well as to provide humidity control. Water-side systems utilize a cooling medium, for example water. Outside and inside air is segregated and cooling is provided through a heat exchanger. The cooling medium is circulated through the cooling towers and/or condensers. (DCD Intelligence b., 2015) 3.4.9.5 In-Rack and In-Row Cooling In-rack and in-row cooling can increase both power density and cooling efficiency in a data center, compared to the conventional raised-floor design (Barroso et al., 2013). For in-rack cooling, an air-to-water heat exchanger is added to the back of the rack so that the hot air is cooled immediately by the chilled water circuit. In-row cooling works essentially the same, except the heat exchangers are placed adjacent to the rack instead of in the rack. Both designs require the chilled water circuit to be brought closer to the servers, which increases costs (Barroso et al., 2013). It is well suited for data centers with high density applications or if targeted cooling is needed. (DCD Intelligence, 2015b) These methods can also be referred to as indirect liquid cooling, as air is still the primary coolant but chilled liquid, e.g. water, is brought closer to the servers (Kleyman, 2016). There are also methods for direct liquid cooling, or component liquid cooling, where the liquid flows directly on to the hot components within the servers (Kleyman, 2016). According to Ebrahimi et al. (2013) and Davies et al (2015), bringing the cooling liquid closer to the servers enables more efficient heat transfers and higher quality waste heat. Direct cooling of processors also increase processor performance and lifetime.

3.4.10 Efficiency Measurements Energy management to increase energy efficiency in data centers offers both decreased power and operational costs, as well as more environmentally friendly operations. There are several metrics for measuring energy efficiency in data center, with different applications. One of the most commonly applied metrics is power usage effectiveness (PUE). It is the ratio of total facility power to IT-equipment power. It reflects the quality and effectiveness of the infrastructure of the data center. Historically, the PUE of data centers has been poor. A study from 2006 showed that most data centers had a PUE around 3.0 (Barroso et al., 2013). This has improved in recent years, and many modern data centers advertise a PUE around 1.2. However, there are several issues related to the PUE metric. First, all PUE measurements do not include the same parameters in the practical use. Also, some operators advertise a design PUE based on optimal conditions, as an aspirational PUE. PUE will also differ depending on operating conditions, and instantaneous PUEs can differ vastly from average PUE. PUEs can also be misleading if measurements have been infrequent. The PUE metric has also been criticized for not always being a fair indication of better energy performance. For example, using an older server with higher electricity consumption relative overhead electricity consumption could generate a better value for PUE. (Barroso et al., 2013) Due to the shortcomings of PUE, and foremost because of its inability to account for energy needed to produce cooling and the recovery of residual heat, Greijer (2010) suggested the measurements Net Power Usage Effectiveness (NPUE). This would account for the energy which is transported to and from the data center, instead of only the power consumed inside the data center. NPUE is by Greijer defined as total net flow of energy, divided by energy consumed by IT equipment. For servers, there is a metric referred to as server power usage effectiveness (SPUE), which is the ratio of total server input power to computational power. Computational power in this case includes the power consumed by the components directly involved in the computation, such as disks, motherboards etc. A modern SPUE should be less than 1.2. The product of PUE and SPUE provides an assessment of the electromechanical efficiency of a data center, and is referred to as total (or true) power usage effectiveness (TPUE). In addition to the above discussed data center efficiency, there is another dimension regarding the efficiency of computing. This will not be further discussed, but considers the value obtained from the energy spent in the computations. (Barroso et al., 2013)

3.4.11 Tier Classification System The design of data centers is often classified using the Tier classification, created by the Uptime Institute. According to the survey conducted at the Data Center World conference, the majority of data center operator applying a data center standard, apply Uptime Institute (Mortenson, 2014). Data center operations are dependent on the integration of several infrastructure subsystems with individual technologies for power distribution, cooling etc. The Tier classification provides actors and stakeholders with tools to adequately identify the expected performance of different data centers.

The classification consists of four standards, I to IV, with definitions for data center infrastructure and corresponding performance confirmation tests. A Tier I data center has the lowest performance expectations as there is no installed redundancy in either power capacity components or distributions paths for power. There is sufficient capacity to meet the needs of

26 the data center, but both planned and unplanned maintenance will require downtime. As classification increase, level of installed redundancy increases. Tier II has redundant capacity components, if there are N active components of a certain type, the data center has N+1 of such components installed. Certain maintenance can be performed with no downtime. A Tier III data center has both redundant capacity components (N+1) and multiple, independent power distribution paths, and the center is only susceptible to disruption from unplanned events. A Tier IV data center has the highest level of redundancy with multiple, independent and physically isolated systems with redundant capacity components. All IT equipment is dual powered, and complementary systems and distribution paths must be compartmentalized. Continuous cooling is required. A failure in any part of the system will not impact the IT equipment, and the system automatically responds to prevent failure from spreading. Thus, the data center is not susceptible to disruption from planned or unplanned events. (Uptime Institute, 2009)

3.4.12 Heat Recovery in Data Centers All consumed electricity in data centers is ultimately converted into heat, which needs to be removed to enable recommended operating temperatures. The potential value of residual heat is partially determined by applied cooling technology and from where the energy has been gathered. As can be seen in Figure 6, higher temperatures are achieved further into the servers. However, as previously stated, the dominant cooling technologies apply CRAC or CRAH units. (Davies et al., 2015). The optimum point for recovery of waste heat in an air- cooled data center would be at the rack exhaust, although this would be logistically challenging (Ebrahimi et al., 2013). It could be easier to capture heat at the return to the CRAC or CRAH unit, allowing for 30 oC to 40 oC, or at the chilled water return with temperatures between 10 oC to 20 oC (Ebrahimi et al., 2013; Davies et al., 2015). Harvesting waste heat through the chilled water system could allow for a greater proportion of the heat to be recovered, but would entail lower temperatures compared to if the heat is recovered from the return air. (Davies et al., 2015).

Figure 6 Typical temperatures of components and processes in data centers. (Davies et al., 2015) Using the chilled water return, heat recovery could be designed to operate independently from the standard cooling system, and thus not affect the overall operation if the waste heat recovery system failed. The data center could be cooled by a CRAC or CRAH unit, and the heat rejected to chilled water for heat recovery. See Figure 7, there are two separate loops for rejecting heat. If one fails, the other one is applied. This has a distinct advantage as minimizing downtime and increasing resilience are important to data center operations. (Davies et al, 2015)

Figure 7 Potential heat recovery configuration. (Davies et al., 2015) Davies et al. (2015) list three other important considerations for configuration of waste heat recovery; the amount of heat that can be recovered and at what temperature, the value of the heat in relation to energy input and the cost of equipment, and the additional infrastructure needed. Liquid has higher heat carrying capacity and convective transfer coefficients compared to air, which permits lower temperature difference between the coolant and the component. Liquid cooling can thus apply higher inlet temperature and increase waste heat recovery at temperatures of 60 oC. Liquid cooling also has the potential to recover heat at different temperatures, which would however increase the complexity of the process. (Davies et al., 2015) 3.4.12.1 Upgrading Low-grade Heat Local waste heat recovery for domestic heating is probably the simplest and most efficient method, as waste heat recovered from data center air between 30 oC to 40 oC can be recovered directly. A disadvantage of local waste heat recovery is however the potentially limited demand, which can be mitigated by a connection to a district heating grid with a larger aggregated heat demand (Davies et al., 2015). However, district heating networks usually require temperatures between 70 oC to 120 oC (Ebrahimi et al, 2013; Davies et al., 2015). Heat pumps can elevate the temperature of low-grade heat by utilizing a two-phase refrigerant. A single-stage heat pump absorbs low-grade heat and upgrades it to a higher temperature, while heat pumps with several stages enable heat recovery at different temperature levels. Multi-stage cycles can improve the Coefficient of Performance (COP) and

29 achieve values around 6 (Davies et al., 2015). According to Davies et al. (2015), a COP-value of at least 3 is needed in order for low-grade heat to be viable for upgrade. 3.4.12.2 Distributed Heat Production A distributed energy system (DES) consists of several components for producing, consuming and distributing electricity or heat. As consumers are usually scattered across a region, is has been questioned whether production of heat or electricity should move closer to the consumers, compared to the large, central units usually applied for heat and electricity production. Söderman and Pettersson (2005) stated that an advantage of DES is that low- grade heat flows can be utilized. By that definition, data centers could be considered part of DES. Other advantages stated were shorter plant implementation times, distributed capital need both in time and enterprises, and shorter distribution pipes for district heating and a more flexible production. (Söderman and Pettersson, 2005)

3.4.13 Examples of Heat Recovery Projects from Data Centers Recent development in the data center industry has brought forward initiatives for heat recovery from data centers. The following section intends to provide the reader with insight in what type of opportunities for heat recovery that has been recently developed. 3.4.13.1 Stockholm Data Parks Stockholm Data Parks is an initiative by Stockholms Stad, a district heating company, a power grid operator, and a provider of dark fiber. The project is described as a long-term commitment to make Stockholm an attractive hub for large data centers which employ heat recovery. The project has gathered and prepared all necessary infrastructure elements necessary for large-scale data centers at different sites in the Stockholm area. The first Data Park will be available for construction in 2017. (Stockholm Data Parks, 2017) Stockholm Data Parks have formulated two different offers for cooling, one which they describe as Cooling as a Service (CaaS), and one in which the cooling is managed by the data center operator. Cooling as a Service is offered for free to data centers with a load above 10 MW, in exchange for the excess heat. For the second offer, when the cooling is managed by the customer, the district heating company purchases the excess heat at a price reflecting the alternative cost. (Stockholm Data Parks, 2017b) Each Data Park will contain a dedicated cooling plant where chillers will provide CaaS for the entire park. When applying CaaS, the excess heat from the data center will be distributed to a production plant in a return pipe, where large centralized heat pumps will generate heat for the district heating network. Stockholm Data Parks offer 99.7% availability for CaaS and assumes the data center will install a redundant cooling system to be used the remaining hours. However, a fully redundant system can also be offered. The cooling is supplied at a temperature of 22 oC. (Fortum Värme, 2017). Stockholm Data Parks has opened for client reservations (Smolaks, 2017). 3.4.13.2 Elementica Elementica is a data center project for a large-scale data center with heat recovery in Stockholm as a collaboration with a data center actor and a district heating company. It is marketed as a large-scale Tier 3 facility. (Elementica, 2017) As a data center for colocation, Elementica offer server space to rent, with the intention to recycle heat to the district heating grid and thereby lower price of electricity during the winter. (Elementica Data Center Construction AB, 2015) Construction of the data center was planned to start in 2017

(Elementica Data Center Construction AB, 2015), but was postponed in September 2016 due to an insufficient level of client reservations (Wendel, 2016). 3.4.13.3 Open District Heating Open District Heating is an established project by a district heating company in Stockholm. Businesses with residual heat, such as data center, recover their heat through a heat pump, and sells it to the district heating grid. (Open District Heating, 2017) The project offers solutions for both heat recovery to the district heating grid via heat pumps, and heat recovery through the district cooling grid. The solution for heat recovery via district cooling mix temperatures from both the supply and return pipe of district cooling, in order to provide a temperature of 14 oC. (Öppen Fjärrvärme, 2017) 3.4.13.4 EcoDataCenter EcoDataCenter is building a data center for heat recovery in Falun. The project is a collaboration with the municipality owned Falu Energi & Vatten AB (FEV). Initially, FEV was to form a joint venture together with EcoDataCenter (FEV, 2017). However, the plan for a joint venture was perceived as too controversial and thus suspended. An approach with more clearly divided roles between EcoDataCenter and FEV was chosen (Norin, 2016) The recovered heat will be utilized for drying wood pellets, and supplied to the district heating grid. The wood pellets factory is normally closed during the winter due to a shortage of heat supply. The data center thus enables increased use of wood pellets in the heat production, which decreases use of fossil fuels. (EcoDataCenter, 2017) EcoDataCenter will offer colocation services and is, as of today, the only Nordic data center with the ambition to qualify for a Tier IV certification. The power infrastructure to the site has been completed, but construction has yet to start. (EcoDataCenter b., 2017)

4 Norrenergi Norrenergi represents the energy company in heat recovery collaborations with data centers. This is a presentation of Norrenergi, its perspective on heat recovery in general, and on data centers as partners in heat recovery. In this analysis, we have combined the literature on district heating and cooling systems with the syntheses from interviews made with Norrenergi representatives. If nothing else is specified, references to Edén and Stymne refer to the interviews conducted in 2017.

4.1 Grid and Production The district cooling system is some 30 years younger than the district heating system and is, despite continuous extension projects, far behind in size. The base production units are large heat pumps, in which Norrenergi currently has an excess of capacity. The heat pumps produce both heating and cooling, enabling natural synergies in the energy system. Both municipalities Solna and Sundbyberg are expected to grow at a fast rate in the coming decades, increasing the demand for district heating and cooling in Norrenergi’s systems.

4.1.1 District Heating Norrenergi has a district heating grid of 190 km parallel piping, in the municipalities of Solna, Sundbyberg, Bromma and Danderyd, see Figure 8 (Norrenergi, 2016). There are two heat production sites, Solnaverket and Sundbybergsverket, and total amount of distributed heat is approximately 1000 GWh per year (Norrenergi, 2017). The main measures of delivery to customers are temperature and the pressure differential. Due to pressure losses in the grid, the delivery pressure in one instant can vary as much as from 8 to 2 bars between different measuring points (Edén). Further from the production sites the pipes are smaller, which means that the dilution from bi-flows becomes more noticeable if a connection is added in a remote section of the grid. To maintain the desired heating effect, the flow from the production site and the central pipes needs to be adjusted according with the new flows, should the temperature of the new flow differ from the current temperature in the specific connection. Depending on dilution factors, the flow or temperature can be increased from the main supply pipe. If the bi-flow is high relative the main flow in the pipe, the heat quality of the delivery may be affected. A large part of Norrenergi’s grid is undersized and does not allow for flow increases during cold temperatures, leaving temperature as the single adjustment option in those sections (Edén). In these sections, the temperature of bi-flows is a critical parameter to ensure enough heat quality and quantity to the end customer.

Figure 8 Norrenergi district heating grid. (Norrenergi, 2016) Heat demand is significantly lower during the summer months, compared to the winter months. Figure 9 displays the amount of distributed heat in Norrenergi’s grid from July 2015 to June 2016.

Distributed Heat July 2015 to June 2016

jul-15 aug-15 sep-15 oct-15 nov-15 dec-15 jan-16 feb-16 mar-16 apr-16 may-16 jun-16 Figure 9 Distributed heat. (Olrin, 2017) 33

4.1.2 District Cooling The district cooling grid provides some 100 commercial and municipal properties and is constituted by 40 km of parallel piping (Norrenergi, 2016). Total annual cooling deliveries amount to approximately 70 GWh. (Norrenergi, 2017) The district cooling grid is shown in Figure 10. The product from cooling production is separated into comfort cooling and process cooling, depending on load profile and demands. The cooling is produced as a by-product from the large heat pumps in Solnaverket, and in dedicated cooling machines with access to free cooling from sea water. The district cooling system works with the same principles as the district heating system, and is regulated through flow and temperature adjustments (Stymne). Norrenergi consider the grid to have high availability, during the summer 2016 there were no unplanned interruptions (Olrin, 2017).

Figure 10 Norrenergi district cooling grid. (Norrenergi, 2016a)

4.1.3 Fuel Mix In 2016, Norrenergi had a fuel mix of approximately 30% residual heat and 15 % electricity, as seen in Figure 11. Additional residual heat in the fuel mix would increase the shares of residual heat and electricity. This addition does not affect the share of renewable fuels, at approximately 96 % for 2016, as the heating oil is used due to its different properties compared to bio oil. However, the usage of biofuels in the system will decrease with the increased usage of residual heat, as the alternative is boilers. The availability of biofuels is

34 considered good and stable in Sweden. Norrenergi buys wood powder from northern Sweden and the Baltic States, and transports it by boat or truck. (Stymne, 2016)

Preliminary Values Fuel Mix 2016

25% 30% Residual heat Wood powder Tall oil Bio oil (0 %) 3% Electricity from hydropower Heating oil 15% Purchased heat

26% 1% Figure 11 Preliminary values fuel mix 2016. (Norrenergi, 2017b)

4.2 Market Development Rydén et al. (2013) conclude that the district heating market has undergone a transformation to become more heterogenous. Stymne says some customers have gone from solely relying on district heating, to combining different solutions, e.g. heat pumps and district heating. These customers have heat pumps of their own, meeting the local demand during most of the year, and buy district heating during the winter. This trend implies a need for new business models for energy companies with district heating systems. By introducing district cooling, energy companies with heat pumps can benefit from the by-product reuse within their own system, by delivering cooling in a parallel system. Stymne expects the cooling market to grow substantially in the years to come, while the development of district heat demand is uncertain. Norrenergi differentiates their products by a customer segmentation, based on the load curve and demand of the properties. For example, the cooling product is sold as either comfort cooling, or process cooling, where data centers would be placed in the second category. Norrenergi has implemented a cyclic business approach, to articulate its new perceived market position. The cyclic business approach reflects the growing interest in prosumers on the district heating market. Edén states that an additional, new role is to manage changes in load and production from suppliers of residual heat, while Stymne concludes that their role has moved to become responsible for system balance, in a heterogenous network of numerous input and output point for energy.

4.3 Servitization Stymne argues that the notion of heating and cooling as “two products” is limiting market penetration. He would rather see a servitization of the whole business. This is in line with the recommendations from Kindström et al. (2015), who stress the importance of a holistic customer view. Working with four heat qualities, Stymne says, increases the opportunities to reach more customers and customize the offers to meet certain demands. It also allows for better utilization of the grid capacity, as utilization of return flows provides additional service

35 offers and increases the efficiency of the heat pumps. Arnell et al. (2012) claim that cooperation with nearby industries or other actors is facilitated by utilizing heat on several temperature levels. The process of servitizing the district heating and cooling systems is only at a starting phase, and Norrenergi is still defining the new role in terms of servitization level. Regarding heat recovery from data centers, the servitization level may stretch from only receiving heat, to owning and operating a heat pump at a data center location (Stymne). Stymne argues that Norrenergi could even be well suited to run a data center colocation business on its existing production sites, pointing out that their operations, maintenance and control organization is competent and already operational. Edén states that the current customer usually controls everything inside the property border, but that Norrenergi’s area of responsibility might be broadened if customer demand changes. The current organization would need to acquire additional competences to run cooling machines in properties. However, expanding the service offer to operate heat pumps on client sites, Edén sees as a possible development, even though they will start competing with the data center suppliers. Norrenergi is a member of the Price Dialogue, discussed in 3.3.3 The Price Dialogue.

4.4 Investments According to Stymne, fuel prices and the price on electricity, are the main drivers for strategic investments. Strategic investment projects in new production may have a 5 to 8-year realization time, and the investment horizon is long-term on such projects. Operating in city areas entails limitations in fuel type for heat production and other environmental considerations. As Norrenergi therefore is limited in their choice of investment for expanding production, they see limited benefits for conducting lifecycle cost analyzes, and therefore uses capital budgeting. Edén adds that each investment is subject for thorough sensitivity analysis to assess the risk. In order for Norrenergi to formulate competitive offers, Edén states that it is important the customer knows the demand before a facility is connected to district heating or cooling, as the investment costs for pipes are linear in regards to dimensions. This means that pipes working on half of its capacity were twice as expensive as they needed to be. Stymne recognizes dimensioning as one of Norrenergi’s toughest investment tasks. Norrenergi is a risk-averse company, with limited possibilities to invest on speculation. Even though many historic investments have proven too small, the budget is set for each case. The only possibility to build in advance, Stymne argues, is when Norrenergi has identified a potential in an area where they are already carrying out extension projects.

4.5 Fuel Mix and Pricing Stymne considers Norrenergi’s fuel mix rather favorable in regards to fluctuations in prices. Their base production comes from large heat pumps, meaning price fluctuations on biomass has a moderate effect on their costs. Also, they are connected to the whole Stockholm system, with a large input from CHP operated by Fortum Värme, which decreases the risk of being too dependent on the electricity market. This means that they are feeling confident in their price development on the heat market. However, Norrenergi primarily compare prices with other district heating companies. Stymne says it is hard to compare investments in districts heating to investments in heat pumps, as the parties tend to value capital differently. The

36 problems in assessing alternative costs sometimes lead to problems in assessing competitiveness of the own products. The pricing is cost-based with competition-based elements, meaning the prices vary with outdoor temperatures and heat availability. Stymne believes Norrenergi has competitive solutions, as they provide a good alternative in both the economical and the environmental perspective.

4.6 Production Planning Norrenergi, Fortum Värme and Söderenergi have a production collaboration for heat production and distribution in Stockholm. This entails that all parties jointly work to optimize heat production to achieve the highest possible efficiency in regards to energy, environmental impact and cost (Norrenergi, 2015). Edén explains a larger share of heat deliveries from residual heat sources requires better information on load curves, to be able to plan for efficient use. Hourly planning is the shortest planning period, and all fluctuation within the hour are handled by accumulators and boilers. With the current production planning, recovered heat will always be prioritized, since the sources can’t be disconnected and often have the lowest production cost. The ordinary heat pumps, however, are only prioritized over municipal waste incineration in periods when the heat pumps are needed for cooling production. This is however unlikely in the long run, as there are normally both cooling machines and accumulators for free cooling. (Edén)

4.7 Risks According to Stymne, the investment cycles are long on production renewal and the prices and taxes on fuels change several times during the life time of a production facility. The heat supply and demand also changes over time. The water treatment plant, which is the main contributor to the heat pumps in Solnaverket, is closing in a few years, meaning Norrenergi must look for new sources of residual heat. Stymne and Edén stress that the future demand and competition are the main risks for Norrenergi. Norrenergi continuously evaluates threats to operations.

4.8 Managing District Energy Systems The quality of district heating and cooling deliveries is dependent on temperature and differential pressure at the specific location. Norrenergi designs the district heating infrastructure for 16 bars and 120 °C, and their customers must construct their facilities for deliveries between 65 °C and 100 °C, usually with some oversizing. Today, the lowest acceptable delivery temperature is 65 °C. The heat demand for temperatures down to 2 °C can be managed solely by regulating flow. For colder temperatures, Norrenergi must increase the temperature in the system. The temperature is increased linearly up to 100 °C, which is reached at around -19 °C outside temperature. (Edén) Stymne adds that they usually lack data on the acceptable minimum temperature deliveries of properties in their grid, meaning that they need to do a local inventory each time a new heat input is introduced into the system. Stymne says Norrenergi is continuously working on decreasing the return temperatures in the heating system and increasing the return temperatures in the cooling system. Each degree Celsius of improvement carries a value in form of higher efficiency in the heat pumps, but also in increased infrastructure capacity. Edén explains temperatures vary depending on building characteristics. Due to the diverse applications of district heating, Norrenergi doesn’t have standardized return temperature requirements for heat customers, while cooling

37 customers have a temperature difference requirement of 10 °C between supply and return flow. Stymne would like for Norrenergi to move away from a two-product portfolio, to offer different heat qualities as a service instead. This, he argues, enables for more applications and a deeper market penetration. They could offer a heat quality of 6 °C on the cooling supply pipe, a heat quality of 15 °C on the cooling return pipe, a heat quality of 45 °C on the heat return pipe, and a heat quality of 70 to 80 °C on the heat supply pipe.

4.9 Data Center Heat Input Today, Norrenergi doesn’t account for residual heat deliveries in their available capacity. This means the deliveries go through whenever they occur. If they were to guarantee deliveries, they would entail a higher value, Stymne argues. If a data center lacks access to district cooling, but is located on the district heating grid, Stymne recommends them to invest in a local heat pump to deliver first-rate heat into the district heating system, as they would probably face investments in cooling machines anyways. Edén estimates that heat pump investments in data centers would usually pay back in a 5-year period. Edén also argues that a heat input on the heat return pipe may not be an economically viable option for Norrenergi, as the return temperatures affect heat pump efficiency. In those cases, he suggests upgrading to first-rate heat for input to the supply pipe. If the data center is located on both the district heating and the district cooling grids, Stymne says the Norrenergi infrastructure is most beneficial. Depending on the temperatures in the grid and the demand, they would be able to choose in which pipe to insert the heat. If the data center can’t deliver first-rate heat, it is usually best to insert the heat into the district cooling return flow. Higher temperatures in data centers give higher efficiency and capacity of the system making optimal use of already made investments, Stymne argues. He says an increase from 15 to 25 °C in the district cooling return pipe would almost double the capacity of the investments. The new pricing model for district cooling is supposed to reflect the value added to the system and provide incentives to increase return temperatures. Usually, Stymne says, the limiting factor for capacity in the system is an imbalance between the supply and return flows, implying that large producers of residual heat should be placed in areas with large flows. Edén adds that the dilution effects in different parts of the grid may sometimes allow for lower heat to be fed into the heat supply line, if the receiving properties have equipment to handle the lower temperatures. Edén says the investments to integrate data centers for first-rate heat deliveries are similar to adding a low-cost base-load facility. The bigger the data centers, the better.

4.10 Synergies Stymne states that there are several synergies to be reaped with data center integration. The already made investments in a large distribution infrastructure and heat pumps can be used to receive heat and deliver it to demand points in the system. Also, Norrenergi has an ambitious organization to ensure continuous operation of the systems. Edén adds that local recovery solutions can’t receive all excess heat form large data centers and that the district heating system has more demand points, even during summer.

4.11 Obstacles Stymne mentions cultural differences as a potential obstacle for collaborations between energy companies and data centers. All interfaces will result in friction, and need work to bridge, he says. Norrenergi doesn’t have any reserve power generation, making it exposed to power outages. Stymne says the Norrenergi district cooling has the same availability as local cooling machines, given that they don’t have any reserve power. Edén says two major obstacles are that Norrenergi often enters the heat recovery discussion at a late stage, when the data center has already invested in a solution. Another problem is the pricing of residual heat, Edén adds. The heat is valued higher when it is locally recovered, than if it is fed into the district heating grid.

4.12 Heat Recovery Limitations Stymne states that Norrenergi has a potential to recover 25 MW during the peak summer, limited by the total demand in the district heating system. Edén says they have the potential to adjust the heat production down to the very last heat pump, should there be enough residual heat in the system. But he adds that a growing share of residual heat entails growing dependence on others, in fulfilling delivery obligations. He says Norrenergi also needs to consider having a too high share of residual heat in an area, and the implications on the total system if some businesses were to move or close.

4.13 Development of New Collaborations Stymne explains Norrenergi is in a knowledge building phase, and that the next step would be to enter discussions with potential partners. In the spring of 2017, Norrenergi launched a collaboration for recovery of residual heat with a data center in Solna. The data center currently recycles the heat locally in the property. When the local heat demand is met, the excess heat is delivered to the supply pipe of the district heating grid. The delivered heat corresponds to the yearly heat demand of 100 stand-alone houses. During the development of the collaboration, Norrenergi has worked with the data center supplier as a point of contact (Norrenergi, 2017c). Previously, the property was a customer for both district heating and cooling, before it chose to invest in a local heating solution. (Stymne) This is an example of the growing diversity in customer relations for Norrenergi.

5 Interview Synthesis This section is a review of nine semi-structured interviews with representatives from different parts of the data center industry and related academia, see Table 2. A compilation of interviewee responses is presented in Appendix C: Interviews with Data Center Industry Representatives. The responses are listed in bullet-points and provide an opportunity to view the empirics in a less edited format. All interviews were conducted in spring 2017, and all interviewees have been given the possibility to review their responses.

5.1 Developments on a Macro Scale The following chapter discusses the macro developments and trends for data centers that can be distinguished from the studied literature and the gathered empirics. Knowledge of the ongoing developments and trends within the data center industry is essential for energy companies that wish to establish successful long-term collaborations with data centers.

5.1.1 Types of Data Centers The interviewed actors seem to agree that there is an outsourcing trend in the data center industry. Lundquist states that both the cloud and the colocation markets are growing. When it is time for an upgrade, companies that previously hosted their IT-equipment in-house will outsource. This development, Lundquist states, is partly fueled by the reduction in energy tax for data centers, which other businesses can’t benefit from. Nilsson states he believed that in- house data centers will disappear completely, with varying time frames for different industries. Several interviewees mention consolidation as a trend, both in regards to actors and data centers. This is partly a consequence of fewer in-house data centers. It may also be in part a consequence of the ongoing servitization trend, discussed later in this chapter, where customers who require IT-services are less likely to invest in the IT-infrastructure and more likely to only buy the IT-services. The companies managing and operating IT-infrastructure thus become fewer in numbers, but larger, as there is an increase in demand for IT. Furthermore, consolidation offers economies of scale. Granström mentions that inability to obtain economies of scale is forcing the smallest actors out of the industry, since they can’t compete in pricing. Edge computing can be discerned as another trend in the data center industry. Dahlgren says the cloud-computing trend leads to large-scale data centers, supported by smaller, local data centers with a capacity of a couple of MW. Closest to the end customers, there might be micro data centers in cabinets at the client site. Fridström states that smaller, local data centers are a prerequisite for the large-scale data centers.

5.1.2 Attractiveness for Investments in Sweden Several interviewees mention Sweden’s low electricity prices, especially with the new energy tax for data centers, as an attractive feature. Additionally, electricity in Sweden is considered very “green”, with low environmental impact and CO2-emissions. Both Fridström and Lindqvist state that being environmentally friendly in Sweden does not come at an extra cost. In regards to the beneficial climate which enables free cooling at low cost, it seems to be an attractive feature for foreign investors.

Another beneficial feature discussed in the interviews is the stability of the Swedish power grid and power supply. Fridström says the tradition of energy-intensive industries and high availability in power supply strengthens the Swedish brand. Additionally, political stability was emphasized as an advantage by both Dahlgren and Fridström. Fridström mentions the high fiber connectivity in Sweden as beneficial. 5.1.2.1 Factors for Data Center Establishments in Stockholm The opinions regarding the availability of the future power supply in high-demand areas such as Stockholm seems to be divided. Lundquist states that it is basically impossible to develop large-scale data centers in Stockholm due to limits in the power supply and that data centers should be built in the northern regions instead, closer to the power production. However, Dahlgren states that, as there are projects for large-scale data centers in Stockholm, it is possible to obtain the required power in Stockholm, but it is an issue of location. There are differing opinions regarding whether the ongoing project for reinforcing the Stockholm power grid is sufficient. Svensson doesn’t think power supply will be a problem, but adds that the risk of competition for power increases as European nuclear power plants are phased out. Lindqvist says some data centers in Stockholm have lost business opportunities due to long lead-times from the power distributor.

5.1.4 Servitization There are trends towards increased servitization for both energy companies and data centers. The characteristics of the servitization trends for these two industries, which could be discerned from literature and interviews, are presented in the following section. 5.1.4.1 Servitization of the Data Center There are different levels of servitization within the data center industry. In section 3.4.1 Data Center Actors, it was mentioned that third-party providers can be further divided into three other segments; colocation, managed hosting and Data Center as a Service. From the interviews, it can be gathered that this is not a clear-cut division, and that several companies have attributes of two or more of these sub-segments. The characteristics of the services provided by a data center are related to the data center’s customer and chosen customer segment. Granström states that the colocation business can be a mix of different services, from only leasing data center floor space and server capacity, to offering hosting services. According to Nilsson, there is an inherent conflict between a high servitization level and the security operations. Depending on the sensitivity of the data, a data center customer might require the colocation or hosting company to own the server, to increase trackability and minimize the risk of data being distributed to unauthorized parties. From the interviews, it can be concluded that there is a trend towards so called pay-as-you-go services. The customers of data centers only want to pay for usage and many are reluctant to invest in hardware. Granström stated that there is a focus on operational expenses within the industry. 5.1.4.2 Servitization of Suppliers to Data Centers The trend of pay-as-you-go services seems to be spreading further up in the supply chain of the data center industry. As data center clients demand pay-as-you-go services with shorter contract time frames, data centers demand the same from their suppliers. Lindqvist and

Dahlgren, both representatives of data center suppliers, work with different models to manage large-scale investments, but agrees leasing of equipment is a possibility. Lindqvist states that suppliers servitize to meet demand for pay-as-you-go services. He also states that it is increasingly common for their company to take on service and operating agreements for the data centers. In the case of recovery of residual heat, data centers may sign an agreement with their supplier to free themselves from responsibility, should the deliveries fail. According to Lindqvist, there might be a conflict in the long-run if data center suppliers would offer overall solutions in facilities they own, as they then will compete with the colocations business. However, there are examples of colocation and managed hosting businesses, renting a complete data center from a supplier.

5.2 Investment Factors The following chapter discusses factors affecting how data centers approach investment. The importance of capital expenditure versus operational expenditure and how data centers evaluate a potential investment are important to consider when approaching data centers with projects for heat recovery. As stated in 3.1.2 Applying a Business Perspective, the success of cases for industrial symbiosis is dependent on economic viability for all involved parties.

5.2.1 Capital Expenditures and Operational Expenditures Interviewees argue that investment costs and capital expenditures constitute major cost items for data centers. Svensson states that the supercomputer was the biggest cost item at PDC. However, as there is a servitization trend within the industry it seems that fewer actors are willing to make the investment in IT equipment. Nilsson states that there is a clear OpEx- focused trend in the entire industry, and that the will to invest in hardware declines. This entails an increased popularity for pay-as-you-go models and decreased contract time frames (Nilsson). Granström confirms a general reluctance to invest in servers and IT-equipment these days. He favors the ongoing trend of delivering services to customers, as it brings new business opportunities, but points out that it brings increased difficulties regarding financing. Even though low interest rates reduce the impact of capital expenses on profit, large server investments are harder to finance compared to when customers own their own equipment. Without the client investment, Granström says they must now handle financing with securities on their own business. According to Nilsson, there is an exception regarding managing sensitive data, which usually entails increased capital expenses for hosting companies. When managing sensitive data, it is important to control the server processing the data. Dahlgren states that a high security level for the facility increases capital expenses and that data centers with less sensitive data usually scale down on security.

5.2.2 Capital Budget or Total Cost of Ownership From the collected empirics, it can be discerned that there is a variation in how data centers consider investment decisions. Both representatives of data center suppliers state that investment costs are very important (Lindqvist; Dahlgren). According to Dahlgren, larger projects look at expected TCO, while smaller projects mainly focus on investment cost. Lindqvist claim that many data centers build their business model on the PUE- value. A proposal for heat recovery with heat pumps increases both the expected PUE-value and the investment costs, and is therefore not attractive in those cases. Colocation actors are more 42 aware of operational cost and are more inclined to work for decreased cooling costs (Lindqvist). According to Dahlgren, the importance of TCO is increasing as higher operating temperatures of the equipment enables more savings. The importance of energy efficiency also seems to differ within the data center industry. According to Dahlgren, energy efficiency is not a top priority for most facilities. For actors with an environmental focus, however, energy efficiency is very important. Lundquist state that EcoDataCenter focus on energy efficiency and claims that even though electricity prices are low, it drives significant cost savings. He also claims that if planned for in advance, an environmentally friendly design does not have to entail increased cost. Taking the same measures after establishment will be more expensive, however (Lundquist).

5.2.3 Investment Horizon According to Svensson, the HPC computer at KTH is depreciated over a time frame of 4-5 years, while Nilsson states that most of their equipment is depreciated during 3 years. Nilsson states that even hosting companies can depreciate IT-infrastructure on a timeline of 10 years. Lindqvist states that many companies claim to have very short time frames for payback of investment, but have 10-year contracts with their landlords. Longer time frames would enable heat recovery projects, usually with a payback of 6-10 years. However, even though the operational cost reductions are appealing, the increased investment costs are usually too hard to bridge (Lindqvist). According to Granström, the ability to take on long-term investments correlates with stable, long-term clients. He claims that they, as a data center, can manage investments with a time frame of more than 10 years, due to their stable client base.

5.2.4 Modularity and Scalability With a servitization trend and a tendency to focus on investment costs when evaluating new projects, it seems logical that the industry would strive for modularity and scalability. According to Lundquist, colocation actors usually build their facilities stepwise and scale up as client base is growing. EcoDataCenter, constructing a large data center with a total capacity of 24 MW, builds in 4 modules of 6 MW each, which are in turn further modularized. When one facility is fully occupied, the next one is built (Lundquist). Dahlgren states that data centers strive to build modularly to avoid part-load operation, and won’t start construction until a certain share of the capacity is booked with signed contracts. For example, the construction start for Elementica, a large data center planned for construction in Stockholm, has been postponed due to a prolonged sales process. So far, the facility has not achieved a satisfying occupancy level. According to Lindqvist, a colocation actor usually never reaches an occupancy level above 85-90 %. The occupancy levels of a colocation actor affect its ability to scale up operations and its utilization level of installed capacity. Both Lundquist and Nilsson name client attraction as a limiting factor for expanding their business. As mentioned above, EcoDataCenter manages this issue by planning for a modular and scalable facility. Lindqvist points out that it is important to data centers to be able to estimate when they will need to expand their facilities, to avoid capacity deficiencies. He also mentions several examples of when new facilities are built and left empty for years. Dahlgren agrees, saying that it may take years to fill up a facility. However, part-load seems inevitable as a high level of rented capacity doesn’t

43 necessarily entail an equally high load. Both Dahlgren and Lindqvist claim that data center customers often only utilize a part of the rented capacity. It is inefficient to utilize equipment and infrastructure on a lower load than what is dimensioned for. However, as the capacity is already rented out, the data centers cannot bring in more customers, even if the rented capacity is not fully utilized. (Lindqvist)

5.3 Availability and Control From the gathered empirics, it can be concluded that high availability and control of the facilities are important factors for the data center business. This has a high impact on the data centers’ attitudes towards possible collaboration partners. Different aspects of availability and control are discussed in the following chapter.

5.3.1 Redundancy Redundant systems are essential for most data centers, as they want to minimize or eliminate possible down-time. Failure of a cooling unit could be devastating for some businesses. It is also desirable to be able to service the equipment without shutting it off (Dahlgren). The level of installed redundancy depends on both the desired Tier certification (discussed in 5.3.2 Tier Classification) and the data center’s client group. A data center aspiring to achieve a Tier IV certification needs to be entirely self-sustaining, see 3.4.11 Tier Classification System. According to Lundquist, the national power system and district cooling can only be seen as alternative systems. The facility must be equipped to handle both power supply and cooling by itself, should it be necessary (Lundquist). Dahlgren states that most internet giants would probably only view district cooling as a possible complement. If the data center is only constructed to meet a lower grade of certification, certain downtime may be acceptable during maintenance (Dahlgren). The chosen level of redundancy will affect investment costs, as investment increases with increased availability (Dahlgren). Redundancy is an expensive feature, and several projects with high levels of security and redundancy have not started construction yet. This could imply that data center clients are not prepared to pay for the top certification levels as there are several middle segment services available to the customers (Fridström). Granström explains that their facility is fully redundant, as all power supply is doubled; double diesel-fueled power generators, power distribution from two directions, double transformers, etc. Their cooling solution consists of three systems; the primary system which they control themselves, a connection to district cooling, and an emergency system with separate piping connected to urban water.

5.3.2 Tier Classification The Tier classification system was explained earlier, in 3.4.11 Tier Classification System, as a way to determine expected performance and availability of a data center. From the gathered empirics, it can be concluded that the perceived value of a Tier classification varies between different actors in the industry. EcoDC markets themselves as “The only Nordic Data Center That Meets Uptime Institute Tier IV Specifications”, implying they see a market value in the Tier certification. Lundquist also declares that a Tier IV certification is their ambition. Granström, however, expresses skepticism towards Tier classifications, claiming that they have missed several important

44 aspects of reliable operations. He says they have a high focus on redundancy, but lack a continuous risk assessment. When they have conducted risk assessments, it has been clear that most data center downtime is not caused by a lack of technical redundancy, but human error or sabotage. Granström also claims that their most security oriented clients don’t value Tier certifications, but instead evaluate data center methods or send their own experts to assess a facility. He adds that Tier certifications are sometimes referred to in a misleading way. Lundquist states that colocation clients often need to be educated regarding operation and risk. This implies that the two different companies, which Lundquist and Granström represent, focus on different customer groups. The first customer group may not be well-informed in data center operation and risk, and therefore perceive a value and sense of security from an official certification. The second customer group is well-informed and with its own perceptions of risks in data center operations, and therefore want to assess the facility themselves. Lindqvist, who takes part in certification processes, states that it is common that data centers don’t live up to the aspired certification.

5.3.4 Risk Assessment Granström declares that it is more difficult to control information flows in large organizations, and that insider threats should be an important part of risk assessment. He also claims that a common problem for data centers is that they lack competent staff to run the facilities around the clock. Another risk factor, which seems to be somewhat neglected, is the environment. Lundquist, states that some units must be above ground, such as power generators and cooling units. If they fail, it doesn’t matter if the rest of the facility is secured below ground (Lundquist). According to Lindqvist, some data centers might have high-security facilities but do not include the environment in their risk assessment. He says the main driver is security, and customers often specify requirements on Tier certification and ASHRAE levels. Granström says that depending on the applied system boundaries, a data center can seem more secure than it is.

5.4 Energy and Cooling Energy and cooling efficiencies in the data center have large impact on the preconditions for recovery of residual heat. This section discusses energy and cooling characteristics of data centers.

5.4.1 Established Cooling Technologies Havtun states that most data centers have some form of air cooling. Some might have liquid cooling to the computers, but no commercial actor utilizes liquid cooling of components. He adds that in air-cooled systems, the air rarely circulates as intended. Lindqvist agrees, saying commonly applied air cooling technologies are extremely inefficient. It is important to keep an even temperature in a data center, and the supplied air needs to be adjusted continuously depending on how intensely the components are operating (Nilsson). According to Nilsson, in-row fans allow for increased precision in cooling and is easier to service. If one part breaks, the entire system does not have to be serviced. This makes the data center less vulnerable (Nilsson). Lundquist states that in-row fans utilize more energy as they operate on a higher RPM due to under-dimensioning, implying larger fans for cooling is a more energy efficient solution. Granström explains that a mitigation tactic for a colocation 45 actor is to cool the data center with air from large fans from below the floor is a simple solution to increase adaptability to differences in customer equipment. Dahlgren states that, regarding initial investment, in-row cooling is more expensive but that it is relatively cheap to build a redundancy with large fans.

5.4.2 Temperatures According to Lundquist, the temperature difference for air-cooled systems can vary between 10 oC to 15 oC while Dahlgren claims approximately 10 °C difference between supplied and returned air. It seems the industry has slowly accepted higher temperatures in data centers. Lundquist declares that some data center actors would rather operate their data centers too cold than too hot, but that the accepted operating temperatures have increased in the last few years to some 22 oC. He also believes the accepted operating temperatures will increase further when the new ASHRAE recommendations arrive. Dahlgren claims that inlet temperatures to cooling system are usually 15 oC, but that some data centers operate with 20 oC inlet temperatures. Havtun states that required inlet temperature to a data center cooling system is 20 to 22 oC and if the temperature in a data center is between 10 oC to 15 oC, the applied cooling technology is outdated. According to Lundquist, the temperatures supplied by district cooling systems of approximately 6 oC at inlet and 11 o C at outlet, are too low. For them it would be suitable with 16 oC at inlet and 22 oC to 24 oC at outlet, and that it would be more beneficial to receive cooling from the return pipe of the district cooling grid. When asked about the applicability of 5 oC to 7 oC inlet temperature and 15 oC to 17 oC outlet temperature, the same is said by Lindqvist. He states that in many cases the supply temperature of district cooling is too low, and in many cases the temperature in the return pipe would be enough. Svensson says that for older systems, utilizing CRAC units and with a data center operating temperature of around 21 oC, the lower inlet temperatures are suitable. However, the temperature difference can be hard to reach, when so close to operating temperature. Encapsulated cooling improves return temperature, as temperature inside the hot aisle is approximately 30 oC. For in-rack, liquid cooled systems, the return temperature could be up to 30 oC, in which case it would be a waste to mix it with 17 oC in the district cooling grid. Svensson adds that condensation might be an issue at low inlet temperatures, especially with liquid cooling close to the equipment. To summarize, Svensson states that the district cooling supply temperature is suitable for older cooling systems, but that higher temperatures might be more efficient. Granström claims the current district cooling temperatures are suitable for their operations.

5.4.3 Liquid Cooling Liquid cooled systems, where cooling liquid is brought closer to the servers and operating components of the IT-equipment, could achieve higher temperatures and be more suitable for heat recovery and increase the efficiency in the cooling process (Havtun). According to Granström and Havtun, liquid cooling of the components could increase the lifespan of components. The foremost reasons for why liquid cooling is not applied in commercial data centers today, seems to be lack of economic incentives and an inherent fear of bringing liquid close to the IT-equipment. Havtun states that, in regards to liquid cooling, the data center industry is both

46 afraid and conservative. Liquid cooling has been developed since the 1990s and Havtun declares that it is at least as secure as current standard solutions. Granström agrees that liquid cooling is something to which data centers must adapt their mindset. Lindqvist says, as a supplier to data center, their sales force is skeptical towards liquid cooling as they do not feel they can guarantee operations. Also, liquid cooled systems are more expensive (Havtun; Lundquist). In data centers with a power density up to 15 kW/rack, where air cooling can be applied, air cooling is cheaper (Dahlgren). There seem to be a lack of incentives to either achieve the higher return temperatures in the cooling system, or increase efficiency of cooling. The HPC computer at KTH PDC utilizes liquid-cooled racks, with a return temperature of water at approximately 30 oC. If component liquid cooling had been applied instead, it could have been possible to achieve a return temperature up to 50 oC. However, when the investment in the new HPC computer was made, there simply were no economic incentives to raise the return temperature to 50 oC (Svensson). Both Lundquist and Dahlgren state that liquid cooling is foremost relevant to a specific market segment; HPC computers, which are used for power intensive applications and achieve higher power density. These types of computers can surpass 40 kW/rack in power density and there are no air cooling technologies today which can manage those levels of power density (Lindqvist; Svensson). Granström confirmed that liquid cooling would be more suitable for heat recovery, as high return temperatures of the refrigerant could be achieved even with lower power densities. Lindqvist believes that it is not the community for heat recovery that will drive the development towards liquid cooling. He claims it will be the server manufacturers. When they start producing liquid-cooled equipment, suppliers of data center infrastructure must adapt (Lindqvist). This is confirmed by Svensson, who says all colocation businesses where customers bring their own equipment need to apply standard solutions. Currently, liquid cooling is not the standard. Granström explains that liquid cooling of components would require a completely different operating organization to manage the increased complexity and risk of the cooling system. It would be difficult to motivate the costs for keeping such an operating organization, to mainly do routine maintenance (Granström).

5.4.4 Power and Heat Density From the gathered empirics, it can be concluded that the power density of server racks for third-party providers is rarely as high as expected, nor anywhere near maximum capacity. Lundquist estimates an average of 6 kW/rack for colocation actors. Nilsson estimates between 5-10 kW/rack, explaining that differences are due to the age of the data center and how much power that can be distributed to the rack properly. Granström claims that their power density is between 2-3 kW/rack or 5-6 kW/rack if the applications in the equipment are composed by them. They did a measurement of all their servers with power supply units of 1.2 kW and concluded that they consume on average 158 W. However, the power supply unit is designed to manage the starting current, which is much higher than the operating current (Granström). Both Nilsson and Lundquist estimate that the power density will increase, but at least Lundquist doesn’t believe in any substantial increase and estimates 8 kW/rack in 5 years. Lindqvist states that in general, data centers don’t utilize even close to the design server power density. Dahlgren explains that colocation actors usually dimension the cooling systems for 10-15 kW/rack, but that actual power density is around 5 kW/rack. Granström

47 explains that there is a misunderstanding within the industry, causing customers to ask for more power capacity than they need. He says that customers look at the capacity of their old equipment, and add safety margins to enable more applications. Nilsson claims a higher power density could be achieved by increasing the number of clients utilizing the server, which could also entail increased revenues per server. HPC-clusters are examples of high power density facilities (Svensson). These are constituted by computers performing advanced research calculations in areas such as turbulence research, fusion research and research on the human brain. The demand for such computing capacity is high, and at KTH PDC there is a queue for accessing the computing capacity. Even though it can be concluded that some data centers have a low power density while others have higher power densities, the interviewees also state that the load curve is flat on both short-term and long-term for colocation actors (Lindqvist; Nilsson; Granström) as well as for HPC computers (Svensson).

5.4.5 Power Usage Effectiveness As explained in 3.4.10 Efficiency Measurements, one of the most commonly applied efficiency measurement in data centers is Power Usage Effectiveness (PUE). Different overheads might be included, and time and continuity in measurement of the PUE may differ. Granström states that the PUE only works if one takes responsibility for the entire energy chain, and that the measurement is not adapted for actors who buy district cooling or apply heat recovery. Heat recovery with electric heat pumps would increase the PUE value, as the increased electricity consumption is not linked to an IT-load. When using the PUE -value, Granström explains that it is very important to define the total system and the system boundaries. Lundquist seems to agree, stating that PUE is an important value, but that it is equally important to be transparent in the calculations.

5.5 Heat Recovery In this section, the data centers perspective on heat recovery is discussed. This perspective can later be applied to the energy company’s perspective (see 4 Norrenergi) to identify possible enablers and difficulties for heat recovery.

5.5.1 General Attitude towards Heat Recovery Among the interviewed actors in the data center industry, it seems there is a positive attitude towards heat recovery. Both KTH PDC and the Datacenter (represented by Granström) already apply heat recovery, although in different ways. It seems that even though actors agree that it is a good idea, and the environmentally friendly aspect is an attractive feature, the main driver is still cost. Granström says that collaborations for heat recovery is an issue of price, but that they at the same time are prepared to pay for their image and values. This is in line with Dahlgren’s statement that some actors may be willing to pay for an environmentally friendly profile. Fridström also states that operating a “green” business is always interesting, but that the main driver is cost. Dahlgren states that large data center actors are not currently interested in heat recovery, and that decreased cost of electricity decreases incentives for heat recovery. Both free cooling and operating cooling machines are cheap alternatives (Dahlgren). Even with decreased electricity costs however, heat recovery increases profit per leased rack space (Lindqvist). Svensson also

48 says that for many colocation actors, Stockholm is an attractive location, and for them, selling recovered heat could be appealing. Nilsson says that they have yet to see any customer requirements for heat recovery, and that an environmentally friendly profile and “green” energy input are taken for granted and not something customers expect to pay a premium fee for. However, he states that there are probably customers who are prepared to pay more for environmentally friendly solutions. Fridström emphasized the importance of pedagogic communication of what heat recovery entails to foreign companies. The concept of receiving payment for residual heat is for example completely unknown to American companies (Fridström).

5.5.2 Servitization of District Heating Systems Dahlgren describes a trend where the ownership of local heat pumps is becoming more differentiated. He says some pumps are owned by data centers, while others are owned by the local energy company. Lindqvist stresses the importance of energy companies partaking in the current servitization of the district energy systems. Both Granström and Lindqvist raise the question of servitizing deliveries as well, making it possible to compete for customers on connected district energy systems, owned by different energy companies.

5.5.3 Configurations for Collaboration Configurations for collaborations can include technical aspects, division of responsibilities and pricing model. This is discussed in the following section, as well as examples of heat recovery which has been encountered while gathering empirics. 5.5.3.1 KTH PDC KTH PDC recovers heat to KTH campus, and the process is managed by the property owner. (Svensson). KTH has its own cooling and heating plant, to which the residual heat is delivered. Residual heat is recovered from both PDC and other buildings on campus. Currently, the different residual heat flows do not have separate piping systems, and the hotter water from the liquid-cooled HPC computer is mixed with water of lower temperature. The heat pump elevates the temperature from approximately 17 oC to 70 oC, to be used for heating purposes (PDC, n.d.). Svensson has previously lobbied to install a separate piping system for PDC, to avoid mixing the temperature qualities. PDC is the single largest heat source for KTH campus (PDC, n.d.) and Svensson approximate its contribution to half of the local demand. The campus grid is connected to the Stockholm district heating grid, from which the property owner can buy heat, when needed (Svensson). 5.5.3.2 Datacenter At Datacenter, the residual heat is pumped through heat pumps and delivered to a borehole heating storage. It is thereafter utilized locally in the building complex, where Datacenter is a tenant. Datacenter, as a tenant, was unable to make the borehole investment, as it would make it impossible for them to move. From this perspective, Granström states that it must be the property owner who owns the cooling solution and makes the investment. It is easier for a property owner to make long-term investments (Granström). The borehole heating storage constitutes the primary cooling system, and the redundancy consists of a connection to the district cooling grid, managed by the property owner. Additionally, there is an emergency cooling system with a separate piping system for urban

49 water. Granström declares that there is a mismatch between the property owner and Datacenter in regards to their perspective on uptime and availability. The one owning the heat pump controls it, and the property owner considers district cooling a complete solution in terms of redundancy. If the property owner had the same requirements for uptime and availability, the emergency cooling system would not have been needed and that investment could have been used elsewhere. When Datacenter procured a cooling system a few years back, the variable cost per kWh was deemed too high for the district cooling option. (Granström) 5.5.3.3 Kraftringen and Max IV Kraftringen is an energy and utility company in the south of Sweden. They supply cooling to, and recover heat, from the research facility MAX IV. As MAX IV requires specially adapted cooling and is situated off the district cooling grid, Kraftringen has constructed a local cooling system. The local system delivers cooling on three different temperature levels and is dedicated to the science center. The heat recovered from the facility is distributed to the district heating grid at 80 oC, which is sufficient for the buildings in that area. The pricing model for the residual heat is formulated in such a way that the compensation for heat is never higher than the value in the district energy system. (Gierow) 5.5.3.4 Ownership and allocation of responsibility The allocation of responsibility within a heat recovery collaboration is dependent on the level of servitization of the cooling system and heat recovery preferred by both the data center and the energy company. The integration level of the energy company could vary depending on if the data center wants to manage the heat recovery and how deep a partnership the two parties want to achieve. Both Lindqvist and Granström state that energy companies are considered stable partners. Lindqvist argues that energy companies have foresight, are municipally owned and have longer business cycles. Granström explains that energy companies are a preferred partner to real estate companies, as an energy company has a 24/7 service organization and an around-the-clock mindset. Energy companies also consist of many technicians and engineers, which is a better match for a data center (Granström). Gierow also states that their experience of managing heat pumps 24/7 is an advantage, when entering collaborations. In regards to responsibility for heat production, data centers are not likely to commit to delivering a certain amount of heat (Lindqvist) and Nilsson states that it would not be relevant for them to commit to heat production, as they have built their business model to deliver a certain amount of capacity and security to their customers. As discussed previously, there is an ongoing servitization trend in the data center industry, with a decreased will to make capital investments. The same trend applies to cooling, where the cooling system can be bought as a service. When a data center leases a cooling machine from a supplier, it effectively buys the service of cooling (Lindqvist). Havtun explains that a data center can either buy the service of cooling, or choose to own a heat pump. If the data center owns a heat pump and sells energy to the energy company, it must modify the system boundaries for the business (Havtun). Some data centers do not want to become district heating producers; in which case they could find another party to operate and manage the heat pump. However, it could be problematic to account for the heat pump as redundancy in the data center’s system, if not controlled by them (Lindqvist). Both Granström and Nilsson state

50 that a heat pump which is not managed and owned by the data center, is not within the data center’s control. Nilsson theorizes that a leasing option on heat pump capacity could facilitate integration. According to Nilsson, district cooling can’t be considered as a redundant system, as it is outside of the data center’s control. Granström says it’s sufficient for them to have only one emergency cooling system within the data center’s control, and that district heating could be the first option. It seems that different data centers have different views on control and redundancy. Both Lindqvist and Svensson mentions that there are cases where the redundancy issue has been managed by a connection to both district heating and district cooling. Lindqvist states that he doesn’t think data centers would let someone else operate their cooling system, as they only write service contracts with trusted suppliers. If the heat pump is kept at the data center, the energy company usually won’t be allowed operate it, regardless of ownership solution. According to Lindqvist, it is the party who achieves the best interest rate that will make the investment. There are examples where the energy company owns the pump installation, a supplier manages the installation, and the data center operates it (Lindqvist).

5.5.4 Compensation Models In the case of both KTH PDC and Datacenter, the data centers still pay for the cooling, even though they supply the other party with heat (Svensson; Granström). Granström explains that if a data center is paid to deliver heat, it is only the difference between the price on heat and the cost for cooling which will be important on a long term. Lundquist reasons that residual heat from data centers is competitive in comparison with other fuels, and that the possible payment for heat recovery from a data center is related to the marginal cost of fuel for heat production at that specific location. Gierow explains that in most of their collaborations, they split the investment and revenues with their partner. Transparency in regards to the calculations are important and leads to a deeper trust in the operating phase as well. Lindqvist states that already formulated agreements are preferred to long processes of formulating new agreements for each project. Dahlgren mentions an example where the system at colocation business automatically calculates whether to operate on the district heating or the district cooling system, depending on electricity prices. (Dahlgren)

5.5.5 Enabling Factors Lindqvist say heat recovery is mainly viewed as environmental projects. Heat recovery is a nice-to-have feature with an environmental perspective, and might become a competitive edge for colocation actors. However, for the data center to make an investment decision, they need guarantees for how much heat they can deliver. It is also important that the system is predictable, and day-to-day variations are not tolerated (Lindqvist). Many companies are interested in achieving a good image, and if heat recovery benefits the total system, it would be an appealing aspect along with payment for the heat (Fridström). The benefits of gaining revenues as well as using heat recovery for branding is also mentioned by Nilsson. Lindqvist explains that he wants the data centers to view heat recovery not as delivering district heating, but as receiving low-cost cooling to mitigate the issue of data centers who do not want to be district heating suppliers. Preferably, the cooling process should eliminate the need for a heat pump, but the district heating systems are built for high temperatures only. (Lindqvist) 51

Formulating a service package deal to data centers is emphasized by several of the interviewees. Svensson explains that for some data centers it is possible to exchange one of the redundant cooling systems for district cooling, but it is important that the energy company can offer a package deal which is operationally reliable from the start. The energy company can offer a couple of standardized solutions if they are cost-efficient and with proof of concept (Svensson). Lindqvist mentions that a reference project would be useful, and that smaller colocation actors would appreciate preformulated offers. For establishment of larger data centers, however, the site should be prepared and partners secured. To achieve collaborations with larger data centers, Fridström states that district heating companies should approach organizations with experience in working with internet giants to benefit from the accumulated knowledge. He adds that other district heating companies could contribute with valuable knowledge as well. For already established data centers, Lindqvist states that a connection to district cooling might be a more attractive offer as it signifies a smaller investment than a heat pump. In general, adapting a data center to heat recovery will be more expensive if done after establishment (Lindqvist; Lundquist; Svensson).

5.5.6 Obstacles Svensson states that one of the largest obstacles to heat recovery from data centers, is sufficient availability and reliability. The energy company needs to understand the data center needs for service availability 24/7. The data center and energy company may also have different views on stable operations and tolerance levels. Lindqvist says data center reluctance to become district heating suppliers sometimes require someone else to take on the responsibility of heat deliveries. Large foreign actors, such as global internet companies, are very set in their own methodology, which makes it difficult to sell them the concept of heat recovery (Fridström). In regards to attracting large foreign data center investors, some of the northern regions in Sweden are years ahead in experience (Fridström). Security and investment costs are other important obstacles to overcome (Lindqvist). Svensson mentions that pricing and interface need to be regulated in a collaboration, and Lindqvist mentions that some energy companies are not transparent enough in how they set their pricing. Nilsson emphasizes that measurement of delivered heat can be difficult. Lindqvist discusses the issue of the high temperatures in the district heating systems, and that the low temperatures received from data centers are better suited for local recovery. According to Havtun, heat recovered at around 30 oC is perfect for local recovery. Granström explains that for them, it was beneficial from an image perspective to recover the heat as locally as possible. There was also an existing business case for both the data center and the property owner; the data center achieved lower cost of cooling, and the property owner decreased its heating costs. As Dahlgren mentions, the current heat pump technology cannot deliver 100 oC heat, and can’t keep up during high demand periods. The high temperatures are a problem when payments for heat deliveries increase with temperature. Excess heat is another obstacle discussed during the interviews. Some data centers have a hard time finding an actor who want to receive the heat in the summer, when heat demand is low (Havtun). Both Gierow and Lundquist mention the issue of managing residual heat when there is no demand.

6 Analysis In this chapter, we match data center integration into Norrenergi’s district heating and cooling systems with the framework on industrial symbiosis. We also present enablers and obstacles for possible collaborations. For each obstacle, we propose a mitigation strategy. The analysis provides building blocks for designing a value proposition towards data centers. The sub- sections are a result of information compilation from literature, interviews and company specific data.

6.1 Synergies for Integrating Data Centers into a Municipal Energy System Clustering data centers and district heating and cooling systems could offer opportunities for resource efficiency. The conclusions drawn by Reichenberg et al. (2011), that municipality owned energy companies could function as anchor tenants and attract businesses to a certain region would be applicable to the definition of industrial symbiosis, suggested by Chertow (2000). A mixed approach to industrial symbiosis is when new businesses, e.g. data centers, are proposed to enter an existing system, e.g. municipal energy system, based on their resource streams. Data center attraction is prioritized by the Swedish government and several governmental agencies work to create internationally competitive offers to the growing data center industry. According to Chertow et al. (2008), a symbiotic relationship can be analyzed from the identified exchange streams and their respective synergetic attributes. They propose three opportunities for exchanges in a symbiotic relationship: 1. By-product reuse. 2. Utility/infrastructure sharing. 3. Joint provision of services.

A stated in 3.1 Industrial Symbiosis, industrial symbiosis is most successful when developed from mutual points of self-interest, and not merely environmental benefits. In the case of data centers as part of a district heating and cooling system, we have located following points of mutual self-interest, aligned with the framework from Chertow et al. (2008). In order to follow the recommendations from Behera et al. (2012), we only discuss synergies achievable by present technology and that the synergies would be implemented in line with current legislation. The points of self-interest are not applicable on all data centers and energy companies, but should be viewed as points of consideration for possible collaborations.

6.1.1 By-product Reuse Heat as a by-product from data center operations could be fed into the district heating system. The heat can’t be fed directly into the heating system, but needs to be upgraded beforehand. Local solutions to upgrade the heat require local heat pump investments. A local heat pump can provide a positive contribution as first-rate heat up to some 80 °C, as stated by interviewees, but will dilute the flow during cold outdoor climate. When the temperature requirement of the district heating system can’t be met, the district heat flow must compensate for the dilution. Norrenergi would not favor recovery on the district heating return flow, as it decreases heat pump efficiency. A central heat upgrade is made in the large heat pumps in Solnaverket, but require access to district cooling at the data center site. Whether local or central, heat pumps produce cooling as a by-product, providing cost-efficient deliveries to the cooling systems. If the heat recovery is local, the solution will cover the cooling demand on 53 site, while facilities with central recovery will utilize the district cooling system. Data centers can receive cooling deliveries from both the district cooling supply pipe or the district cooling return pipe, depending on the capacity of the local cooling solution. Cooling data centers with the return flow increases the capacity of the whole system and the efficiency in the heat pumps. The elevated temperatures in modern data centers enable them to accept the return temperatures in the district cooling system.

6.1.2 Utility/Infrastructure Sharing Shared access to infrastructure and utilities stems from the common interest in the shared by- product, i.e. heat or cool, as well as an interest in increased economic and system efficiency. Norrenergi has an established infrastructure for distributing and producing both heating and cooling to an aggregated demand. Additionally, Norrenergi has available capacity in its heat pumps. Recovering heat through the district cooling and heating system entails better use of existing infrastructure as well as increased opportunity to utilize heat during low-demand periods. If a data center supplies heat via a heat pump to the supply line of the district heating grid, the local heat pump signifies a part of the shared infrastructure, as its ability to produce heat of desired quality affects both parties. Delivery of recovered heat via a heat pump would add production in the specific node and decrease the heat production in the central units. The added production could compensate for loss of temperature and pressure further out in the grid, depending on data center location. For the data center, delivery of heat to the supply line would entail an additional revenue stream. If residual heat is recovered to the return pipe of district cooling, the temperature of the returned water is increased, thus increasing the efficiency of Norrenergi’s heat pumps, as explained by Stymne (2017). As stated from several of the interviewed actors in the data center industry, the return temperature of the district cooling grid could be a suitable match for supply of cooling to the data center. A possible integration solution, where the data center both retrieves cold from and delivers residual heat to the return pipe, would grant higher efficiency and increase distribution capacity. If a data center is located within Norrenergi’s facilities, the shared infrastructure will not be the district heating and cooling grids, but instead the infrastructure utilized by both the data center and the energy company on site. Shared infrastructure could be for power supply, cooling, and technical equipment for control functions. Locating a data center on Norrenergi’s production site enable direct access to heat input for Norrenergi, and opportunities to influence the design of the data center cooling solution.

6.1.3 Joint Provision of Services Collaborations for heat recovery from data centers to energy companies enable synergies through joint provision of services. As the gathered empirics show, both parties have service organizations to enable 24/7 operations, unlocking synergies in operation, maintenance and security. The parties have similar needs for monitoring operation, conducting maintenance and managing security without interfering with operations and production.

6.1.4 Other Synergies With a renewable electricity production, the heat pumps, local or centralized, have less environmental impact compared to boilers for fossil fuels or waste incineration plants. From a system perspective, one of the contributions of heat recovery is the reduced heat emissions. The data center industry is known for its growing energy consumption, and businesses trying to brand themselves as eco-friendly have an opportunity to enhance their environmental profile, while lowering the operation costs. Norrenergi only offers services produced with electricity certified to have a high share of renewable energy input. The exposure to price fluctuations decreases for data centers, as the price for district energy services is hedged with a combination of other fuels. For the energy company, a distributed production means less risk for major production disturbances.

6.2 Enabling Factors for Heat Recovery from Data Centers For heat recovery to be feasible, there needs to be a forum for desired services exchange, and the quality of the input must be favorable.

6.2.1 Demand for Heat Recovery Solutions From the interviews and the industry fair, it can be concluded that there is a rough consensus within the Swedish data center industry about which market characteristics makes Sweden attractive to data center establishments; environmentally friendly energy, low cost for electricity and beneficial climate. Therefore, Sweden is experiencing a rapid growth in the data center market. The data center industry is gaining attention as a growing energy-intensive industry, increasing the impact on the local energy systems. With increased attention on an energy market in development, the demand for resource-efficient solutions is high. It has been pointed out in trend reports (BCG, 2016; IDG Connect, 2016) and the Data Center Risk Index (Cushman & Wakefield, 2016), as well as interviews that the data center market is in a consolidation phase, where in-house data centers are being outsourced to large colocation businesses. Alongside colocation, large Internet companies are expanding with regional hyper-scale data centers, supported by small, local edge data centers in the cities. The largest data centers are usually placed in rural areas with a low heat demand to meet, while some colocation providers seek proximity to their customers and consider city-locations. An existing local heat demand is a definite prerequisite to heat recovery, while most data centers can operate continent-wide. The heat recovery solutions offered by city-located energy companies like Norrenergi and Fortum have spurred an interest among environmentally conscious data centers, who are interested in innovative eco-friendly solutions and a green branding. Some data centers are less interested in heat recovery, claiming that already efficient processes don’t have the same savings to make. Some regions, with limited possibility to recover the heat have no interest in promoting the possibility, but may stick to promoting their good climate for free cooling solutions instead. Heat recovery projects are still considered a foremost environmental move, but there are economic profits to reap as well.

Depending on configuration, the data center can take different roles. According to Behera et al. (2012), potential collaboration partners should be chosen bases on their willingness to collaborate in synergies. However, we argue that the solution for heat recovery could be

55 adapted to the data center’s attitude. If committed to delivering first-rate heat, the data center can become a district heating supplier, while a connection to the district cooling system can prove a cost-efficient alternative cooling solution and thus not require any further commitment.

6.2.2 Servitization of Heat Production in District Energy Systems Integration of data centers require the energy companies to re-evaluate their role in the energy system. Allowing for prosumers in the district heating system means a move towards a more flexible market. Previously discussed theory has stated the need for energy companies to update their business models and develop more complex value propositions with integrated products and services. Norrenergi has started its transition to viewing its district heating and cooling systems as an eco-system of many input and output nodes. In this eco-system, the input nodes may be owned and controlled by various players, while Norrenergi is responsible for system balance and distribution to the output nodes. Already, the customer base has shifted character, as some customers use district energy services in combination with other solutions. To stay competitive, Norrenergi has recognized the new trends on the market, and developed complementary services and more flexible pricing.

6.2.3 Product and Service Offers It can be concluded from the empirics that pre-formulated product and service offers would be beneficial to achieve heat recovery solutions in data centers. Increased integration of product and service offers has also been discussed in 3.1.2 Applying a Business Perspective. The offers need to be based on the data centers demands and the design of their operations. As explained in 5.3 Availability and Control, data centers have high demands for availability and security. Heat recovery solutions need to be formulated to mitigate these demands for operational reliability. The conditions for establishment of heat recovery solutions differ between already established data centers and data centers under development. For already established data centers, introducing heat recovery as an ex-post investment becomes more expensive compared to if it is planned before construction. An already operational data center will need to adapt its current operations and infrastructure to recover heat, introducing another layer of redundancy on top of the existing. As stated by Lindqvist (2017), connecting an existing data center to district cooling is a smaller investment than a heat pump, and lowers the threshold to enter collaboration. For new development of data centers, requirements vary depending on data center size. Large data centers, foremost Internet giants, demand short time to market. They often use organizations providing package deals, where infrastructure partners and building permits have already been secured. To secure heat recovery solutions with large data centers, energy companies need to take part in existing collaborations and organizations for data center investments. For all project offers, it’s desirable to provide pre-formulated agreements on price and availability. Preferably, solution design and required permits should be provided when approaching clients. As stated in 5.1.4.2 Servitization of Suppliers to Data Centers, some data centers chose to hire data center suppliers to manage heat deliveries from heat pumps. Another option is for the energy company to formulate integrated product and service offers together with a data center

56 supplier. With a partner that is well-known within the data center industry, this can increase the perceived reliability and attractiveness of the offer. As stated in 3.3.1 Competitive Business Models for District Heating and Cooling¸ some energy companies use networks of subcontractors to achieve holistic service offerings. Tsvetkova and Gustafsson (2012) discussed, in 3.1.2 Applying a Business Perspective, modularity in solutions offerings as a measure to mitigate for adaptations to local conditions in. The compensation model for delivering heat needs to be carefully designed by the energy company, to avoid undesired investment incentives. If the pricing drives heat pump investments for data centers located on the district cooling grid, the energy company risks losing local heat customers to the data center solution. Furthermore, they lose the data center as a cooling customer. The preferred solution for energy companies is to receive first-rate heat from data centers without access to district cooling, while receiving low-grade heat from data centers connected to the district cooling system. As discussed in 5.1.4.1 Servitization of the Data Center, the increased servitization trend has led to challenges in data center financing, meaning energy companies often have better opportunities for making long-term investments. The compensation model is more thoroughly discussed in Appendix B: Compensation Model for Recovered Heat. Proof of concept or reference projects has also been highlighted during the interviews as a tool for attracting both smaller and larger actors in heat recovery and help visualizing the benefits. Norrenergi has launched a pilot project for heat recovery from a data center, to serve as a proof of concept for future collaborations.

6.2.4 Heat Quality Traditionally, data centers have appreciated cold temperature in their facilities, resulting in high cooling costs and low quality heat in the outlet air. New standards for IT-equipment and better cooling infrastructure have enabled indoor temperatures to rise above 20 °C and outlet air temperatures of over 30 °C, in some cases. As stated in both 3.4.9.1 Temperatures and 5.4.2 Temperatures, a 10 oC to 15 oC difference between supply and return air is common for efficient air-cooled systems. Dahlgren (2017) explained that if a data center aims for a higher outlet temperature, it needs to accept a higher inlet temperature. The elevated temperatures contribute to lowering cooling costs and enable free cooling to a higher extent than was previously possible. The higher temperatures also open for new district cooling and heating solutions. If the data center chooses to become a first-rate heat supplier, less heat pump power is needed before feeding the district heating grid. An interesting possibility is to cool data centers with the district cooling return flow, resulting in higher capacity for the district cooling system and higher efficiency in the heat pumps. Feeding the heat into the district cooling return pipe requires no local heat pump and connection costs are similar to a normal district cooling solution. Component liquid cooling technologies have the possibility to managed increased data center temperatures, enabling outlet temperatures around 50 °C. Liquid cooling directly on the components has the advantage of being efficient in both ends of the heat recovery, reducing losses in turbulent air flows and large heat exchangers. These technologies are still considered immature and inconvenient for most commercial data centers. Both equipment and installation are more expensive and maintenance requires a more competent staff. Colocation businesses are already trying to minimize operational risks in server changes, and want to 57 keep the routine maintenance as simple as possible. In 5.4.3 Liquid Cooling, we see that industry representatives think liquid cooling will only be relevant to colocation businesses if the IT manufacturers switch from air cooling as standard solution. Air cooling is sufficient for most facilities, as power densities in standard data centers are rather low. HPC clusters, seeing their heat densities increase rapidly, may deem liquid cooling relevant in a matter of a few generations.

6.3 Obstacles to Successful Heat Recovery Heat Recovery from data centers is a new phenomenon, compared to heat recovery from traditional process industries. The characteristics of the data center industry pose new challenges in collaboration design.

6.3.1 Culture In 3.2 Experiences from Projects for Heat Recovery in Process Industries, we see that lack of knowledge and understanding of each other’s businesses was one of the main difficulties with collaborations between energy companies and industrial firms. This view was endorsed by interviewees from the data center industry, stating that energy companies lack in understanding of data center operations. The lack of understanding seems to be foremost concerning temperatures of supplied cooling and availability requirements for the cooling and heating grids. Energy companies are perceived as reliable companies in other aspects. One interviewee mentioned communication problems on tolerance levels leading to poor quality in the cooling deliveries in a recent project. Additionally, McEwen (2015) stated that different operational goals between industries and energy companies could hinder collaborations. Communication and transparency are important mitigation tools to address the lack of understanding, as stated in 3.1.2 Applying a Business Perspective. Both Norrenergi and the data center industry are characterized by high standards on operations and 24/7 service requirements. Although they share similar operations regimes, the different organizations are constituted by different competences and skills. If an energy company wants to increase the competitiveness of a heat recovery project, they should increase knowledge of data center requirements within their service organization. Perimeter security ambitions vary with client base between different colocation actors, but is usually a concern. District energy companies have well-developed risk assessments for internal risks, but need to improve the awareness of external threats to match data center demands. Another culture related issue is the data centers’ reluctance to become district heating suppliers. Industry reluctance to stray from the core business by recovering residual heat has been previously discussed in both 3.1 Industrial Symbiosis and 3.2 Experiences from Projects for Heat Recovery in Process Industries, and was also discussed during an interview with a data center supplier. As heat production is not part of data centers core business, related investments can be more difficult to motivate. It would be easier to motivate an investment if the provided service minimizes the data center’s commitment and responsibility. Some data centers have already created these types of solutions, hiring data center suppliers to manage heat deliveries. Connecting to district cooling as a measure to decrease operating costs, is closer to core business and easier to motivate compared to a heat pump investment for external deliveries. As stated in 5.4 Energy and Cooling, actors within the data center industry have diverse attitudes to the importance of energy efficiency. Some data centers prioritize energy efficiency, while others don’t experience sufficient incentives to increase efficiency. In 58

3.4.3 Attractiveness Factors for Investments in Sweden, the lowered energy tax is pointed out as a reason for decreased incentives for energy efficiency. Different approaches to energy efficiency could be an issue during collaborations between energy companies and data centers. To mitigate the issue, it will be important to locate the factors that could bridge the issue, such as economic benefits or a more efficient cooling process. The most commonly used measurement for energy efficiency in the data center industry, PUE, is considered flawed (see 3.4.10 Efficiency Measurements). It is highly dependent on the applied system boundaries, and is used differently by different actors (see 5.4.5 Power Usage Effectiveness). The usage of PUE could disfavor heat recovery collaborations, as PUE increases with the use of electric heat pumps. As discussed in 3.2 Experiences from Projects for Heat Recovery in Process Industries, case studies of industrial heat recovery stress that having an employee within the industry with expressed responsibility for energy issues and energy efficiency, is a success factor. During the interviews, a data center supplier representative stated a lack of dedicated energy responsibility is a problem when setting up new solutions. However, the interviewed representatives from two colocation actors clearly expressed an energy responsibility allocation in their organizations. Both colocation actors had also made informed decisions regarding energy efficiency and heat recovery. Thus, the role for energy responsibility within the data center industry is not standardized. However, it seems that a clearly dedicated responsibility for energy issues and energy efficiency is beneficial for heat recovery projects also within the data center industry.

6.3.2 Availability High availability in operation is an essential requirement for most data centers, as stated in both 3.4.2.3 Availability and Control and 5.3 Availability and Control. Few data centers can allow for any downtime at all, which they avoid by building redundancy for all operationally essential infrastructure. If an energy company takes part in the cooling solution, either by supplying district cooling or receiving heat to the district heating grid, most commercial data centers will need to install redundancy in in their local solution, as energy companies can’t guarantee 100 % uptime. The pricing models should be formulated to incentivize the data centers to operate their systems for heat recovery during the majority of operating hours. Maintenance of the grid should be communicated beforehand, to meet the need of control from data centers. Norrenergi should ensure that they properly communicate their high availability of district cooling.

6.3.3 Investments Capital budgeting is a popular method among data centers for evaluating projects, as stated in 5.2.2 Capital Budget or Total Cost of Ownership, meaning capital expenditures have an elevated importance compared to operational expenditures. Both Greenberg et al. (2009) and Barroso et al. (2013) discuss the importance of capital expenditure to data centers in their previous studies (see 3.4.8 Cost Analysis). Larger data centers may calculate lifecycle cost or total cost of ownership, but the main issue is often initial capital investment. With the ongoing trend in servitization, data center customers prefer renting server capacity instead of buying servers. Data centers thus need to invest more in IT-equipment which, according to one interviewee, creates issues with financing. As mentioned in 3.4.2.6 Capital Expenditures vs. Operational Expenditures, the CEO of a colocation actor stated financing as very important,

59 and mentioned the company’s financially strong owners as an advantage. Additionally, both representatives of interviewed data center suppliers stated investment costs as an important consideration. Heat recovery projects in existing data centers entail additional investment costs, deterring potential investors even though it decreases their operating costs. As explained in 3.4.8 Cost Analysis, investment horizons for IT-equipment is usually short; around 3-5 years (Barroso et al., 2013; Greenberg et al., 2009), while around 10 years for infrastructure (Barroso et al., 2013; Rasmussen, 2012). The interviews reflected that the pre- disposition towards longer investments might be dependent on the applied business model. Shorter time frames for client contracts decreases ability to manage long-term investments, while longer contract time frames increase it. This is connected to the previously discussed OpEx trend in the industry, and the decreased willingness to make large investments. Mitigating conditions as long-term rental agreements and stable client base can extend the investment horizon. In 3.4.9.2 Power Density we mentioned that Ebrahimi et al. (2013) describe a direct proportionality between data center cost and floor area, and that server manufacturers therefore strive to increase power density of their modules. This relationship has not been confirmed by the interviews. No interviewee has mentioned floor area as a limiting factor for growth, or mentioned it as a cost-increasing factor. Lundquist (2017) explains that data storage, which is expanding, is much more space-efficient with the new SSD technology. To achieve heat recovery collaborations with data centers, energy companies need to formulate offers which mitigate the reluctance against the initial investment. Per discussion in Literature and Theory, a key component for investing in common infrastructure is that the risk taken by each party is reflected in the profit from the investment. Energy companies could offer to make the investment themselves, providing only a cooling service to the data center, transforming capital expenditures into operational expenditures. If Norrenergi choses to enter partnerships with data center suppliers, they have the option to move the responsibility for client management downstream. A partnership with a data center supplier could also mitigate some of the cultural issues and increase the credibility of the service.

6.3.4 Occupancy Levels Interviewees witness of long lead times to reach acceptable occupancy levels and, even with a sold-out facility, low utilization of capacity. To improve efficiency and increase return on investment, the data center industry is becoming more modularized and scale up investments as client base expands. The scalability of the data center has improved, and businesses can become operational without knowing its final size. While the risk of over-dimensioning the data center infrastructure has diminished, the risk of over-dimensioning the district energy connection persists. The risk for the energy company seems elevated, but the change lies within the improved information flow and not the operations. If the data center declares an intent to scale up business, the energy company can take informed decisions on whether to dimension for only a part of or all the expected final capacity. Without the modular approach, data centers can only declare a target final capacity. The utilization of installed capacity differs substantially between data centers. Infrastructure is always dimensioned for startup power demand for a fully booked facility while the operating power, in combination with a poor occupancy level, is far lower. The expectation on heat

60 deliveries must follow the actual usage of IT-equipment, requiring a close dialogue between energy company and data center. Power and heat density becomes a factor if dimensioning a data center with limited space. Depending on usage area, the average heat density in a data center ranges from a single kW/rack in in-house data centers to several hundred kW/rack in HPC computers. Interviewees say 2 to 10 kW/rack for colocation businesses, depending on clients and equipment composition. One interviewee states colocation facilities are usually dimensioned for 10 to 15 kW/rack, but the usage is usually around 5 kW/rack. The energy company interested in recovering the heat, might be disappointed in the low turnout if they dimension the heat recovery for the total data center capacity.

6.3.5 Temperature Tolerance and Customized Solutions An operational problem for first-rate deliveries into the district heating system is the issue of heat pump output temperatures. The value of the heat increases as outdoor temperatures drop and the temperature in the system is elevated, but heat pumps rarely deliver above 80 °C. If situated in a disparate part of the district heating grid, the main flow may be too small to compensate for a diluting bi-flow from a data center. Large data centers would preferably be located close to a main pipe in the district energy system, to minimize dilution. New property standards and infrastructure often allows for flexibility in delivery temperatures, moderating the importance of dilution. However, Norrenergi has an information deficit on the property standards in their system. When investigating the possibilities of lowering supply temperature in an area, an inventory must be carried out to secure quality. District cooling simplifies integration but, for Norrenergi, the district cooling grid development is some 30 years behind the district heating grid and is not available at all locations. For substantial data center projects, customized solutions can be an option; Stockholm Data Parks has a local cooling system with large heat pumps, meeting local cooling demand and delivering heat into the Stockholm district heating system; In Lund, the synchrotron radiation facility MAX IV has a local cooling grid, connected to the district heating grid. Customized solutions can be optimized for the specific cooling demand and be dedicated to the local businesses. Data centers usually have a flat load curve over days and seasons, making them well-suited for base-load deliveries into the district energy system. The colocation businesses and host representatives in this project have a diversified client base, reducing fluctuations in production. Norrenergi uses heat pumps as base load during all seasons, making the production comparatively flexible. The heat pumps can be controlled and, if necessary, shut down completely when demand is met. Thus, the maximum input from residual heat is set by the total system heat demand, which drops to 25 MW in the peak summer.

6.3.6 Accessibility to Data Center Market Usually, the infrastructure of a data center, such as cooling solutions and power supply, is hard to change during a data center life time. There are small possibilities to interfere with the operations, and instead heat recovery solutions must be added to existing infrastructure. Norrenergi concludes that they often enter the process too late, missing the design phase and the possibility to optimize the local solution for heat recovery.

6.4 SWOT-analyses of Examples of Configuration The different configuration types symbolize possible ways to integrate data centers into a district heating and cooling system. Each configuration type has certain characteristics, affecting the respective business opportunities. As discussed in 2.2.1 Case Analysis Structure, SWOT-analyses are a valuable tool to support decision-making processes and help map a strategic approach. The SWOT analyses in this chapter concretize discussed synergies, enabling factors and obstacles to each individual collaboration, to help evaluate Norrenergi’s role in each case. The examples of configuration are visualized in Figure 12.

Figure 12 Configuration possibilities for connection of data centers to district energy systems.

6.4.1 SWOT configuration A – Colocation on Production Site The energy company offers colocation services to data center hosts on the energy company’s production site. Configuration means sharing of space, infrastructure and services. Operation costs could be decreased with local heat recovery at the central heat production site. The facility can be optimized for high energy-efficiency. The SWOT-analysis for configuration A is presented in Table 3.

Table 3 SWOT-analysis of configuration A. Configuration A Strengths Weaknesses • Proximity to traditional heat • Large initial investment. production diminishes dilution • Lack of competence. effects in the system. • Difficulties in attracting a customer • High integration level. base to serve heat production. • Utilization of existing heat pump. • Establish relationships with new • Synergies in production and services. suppliers. • Colocation sales. • High demand on communication and • Possibility to customize cooling transparency. system and heat recovery. Opportunities Threats • Branding opportunity – innovative • Long lead time to fill up desired thinking. capacity. • Liquid cooling could provide high • Competition with established temperatures and high heat densities colocation actors. for optimal use of space. • Uncertainty in customer heat density.

6.4.2 SWOT configuration B – District Heating and Cooling The data center is located on both the district heating and the district cooling grids. The energy company provides heating and cooling services for local facilities. Low investment need in data centers. Heat is recovered and fed into the most suitable flow. Cooling supply temperature is flexible and can be optimized for local demand. The SWOT-analysis for configuration B is presented in Table 4.

Table 4 SWOT-analysis of configuration B. Configuration B Strengths Weaknesses • Maintain cooling and heating • Some data centers will need to make business. double investments to keep desired • Four different heat qualities enable redundancy. more services and connection • Difficult to dimension connections flexibility. for modular data center growth. • Complex pricing dynamics. Opportunities Threats • Connection to cooling return flow • Customer chooses to utilize its own increases efficiency and capacity of cooling equipment during expensive the system. periods. • Low return pipe flows can be • Local collaboration with property compensated by mixing in cooling owner puts recovered heat at market supply. value1. • Green branding.

1 See Appendix B: Compensation Model for Recovered Heat for discussion on compensation model. 64

6.4.3 SWOT configuration C – Distributed Heat Production The data center has no local demand to meet, and is connected only to the district heating system. The district energy system provides possibilities to distribute excess heat. Ownership and operation of local heat production can be adapted to desired involvement and investment will. The SWOT-analysis for configuration C is presented in Table 5.

Table 5 SWOT-analysis of configuration C. Configuration C Strengths Weaknesses • Dedicated heat deliveries. • Heat dilution during peak demand. • Distributed heat production. • Difficulty to dimension for modular data center growth. Opportunities Threats • Data center suppliers could manage • Reluctance of data centers to invest heat deliveries. in heat pumps. • Standardized designs and solutions. • Reluctance to commit to a certain • Possibility to servitize through local level of heat deliveries. heat pump investment. • Green branding.

6.4.4 SWOT configuration D – Distributed Heat Production with Local Load The data center is connected to district heating and a local load. The district energy system provides an aggregated demand to receive excess heat, when local demand has been met. Ownership and operation of local heat production can be adapted to desired involvement and investment will. The SWOT-analysis for configuration D is presented in Table 6.

Table 6 SWOT-analysis of configuration D. Configuration D Strengths Weaknesses • Distributed heat production. • Heat dilution during peak demand. • Difficulty to dimension for modular data center growth. • Uncertain heat deliveries. • Complex pricing dynamics. Opportunities Threats • Data center suppliers could manage • Local collaboration with property heat deliveries. owner puts recovered heat at market • Standardized designs and solutions. value2. • Possibility to servitize through local heat pump investment. • Green branding.

2 See Appendix B: Compensation Model for Recovered Heat for discussion on compensation model. 66

7 Discussion In this chapter, we bring forth recommended strategies to approach collaborations, discuss the results, and offer some suggestions for future studies.

7.1 Recommended Strategies From the analysis, we have singled out a few strategies for energy companies to use in the approach towards data centers with potential to deliver heat into the district energy system.

7.1.1 Know the Potential Partner We recommend to carefully study the mindset of potential heat recovery partners and to tailor the pitch accordingly. There are at least three major categories of data centers. Each category is, in turn, diversified with niche services within availability, security etc. How prone a data center is to integrate the business with external organizations depends on their organizational maturity, how risk averse they are and their price sensitivity. The energy efficiency is one of the most discussed factors when operating data centers, which means that the right offer for the right organization has high potential of becoming subject for consideration. However, as becoming a heat supplier is not part of the core business of data centers, many owners aren’t susceptible to an offer of heat deliveries to the district energy system, especially those who aren’t familiar with district energy systems. The economic factors of a project may be a second priority, and they are inclined to reject even projects with short payback times. Those owners would be more inclined to accept an offer of reduced cooling prices instead.

7.1.2 Target Third-party Providers We recommend to target third-party providers on the existing grids for heat recovery projects. Forming heat recovery collaborations with global internet providers requires experience in how to attract and retain these actors, and usually involves a cluster of governmental and regional organizations. The growing third-party provider segment could provide stable partners. Small or medium third-party providers are easier to establish collaborations with, due to limited size of the organizations.

7.1.3 Build Alliances and Product and Service Offers We recommend to build alliances and offer prepackaged solutions. Alliances increase interface between the energy company and data centers. Potential partners are data center suppliers, industry and trade organizations, and energy and power companies. The partners not only contribute with products and services, but also their brand and knowledge of the industry. With or without alliances, standardized integrated product and service offers should be composed to decrease effort and friction when approaching a data center with heat recovery solutions. The integrated product and service offers should include propositions on how to manage desired redundancy and possible heat quality options. If composed together with partners, the offer could include power deals and infrastructure service and operation.

7.1.4 Use District Cooling to Prevent Local Substitutes We recommend to use district cooling, and preferably the return flow, to cool and recover heat from data centers. The option of recovering heat in the district cooling return flow can optimize utilization of grid investments by increasing capacity of pipes and efficiency in pumps. Mixing the supply and return flows allows for customized heat quality deliveries, tailored to the specific local needs. Another advantage of heat recovery in the district cooling system is an increased client base for district cooling and a retained client base for district heating. The synergies entailed by heat recovery increases price competitiveness compared to substitutes, like local cooling machines, as production price is low and no large investment is needed.

7.1.5 Explore Possibilities of Ownership of Distributed Production We recommend to explore the possibilities to invest and own heat production at the data center site. Where no district cooling is available, the heat is preferably recovered on the district heating supply flow, after being upgraded with a local heat pump. A heat pump investment is outside of regular data center planning and needs special considerations from the data center owners. The short contract times and investment horizons, in combination with capital budgeting, increases the threshold for investments in reducing operational costs or environment. If the energy company, with a longer investment horizon, takes on the investment, the threshold is lowered. Servitization of data center suppliers is already well under way, and proposing a joint project in investing and operating a local heat pump, wouldn’t be far from their usual business. No matter who makes the investment, a success factor is to let the profit made by each partner reflect the individual risk. This means that a data center with low stakes, should expect less returns from the collaboration, even after the project has paid back.

7.1.6 Focus on Cost-efficient Cooling We recommend to primarily offer cost-efficient cooling to all data centers. If the data center is located outside the district cooling grid, we recommend individual compensation for heat deliveries, based on the local market conditions. Regarding data centers with local heat demand, we recommend to consider local investment to avoid competition for heat customers. Local heat pump investments can be made by either party, but the compensation model needs careful consideration. If the data center invests in a heat pump and is connected to a local heat load, they can deliver heat at market price to the local demand, before delivering to the district energy system. Local deliveries effectively mean lost heat customers. If the data center isn’t connected to a local heat load, the energy company has a better position and competes only with alternative cooling substitutes for the data center. With the same payback time and cost of capital, both parties are equally likely to consider the investment. However, the usually longer investment horizons for energy companies allow for a lower compensation level to make the investment profitable.

7.2 Reflections on Results Both literature and interviews have been aimed to distinguish a current state of the data center industry. Our results contribute to the academic body of knowledge by acknowledging the applicability of the industrial symbiosis framework on the emerging concept of data center integration into district energy systems. Industrial symbiosis has been criticized for not accounting for the economic aspects of industrial collaborations. Therefore, our interviews have been focused on discussing the economic incentives and strategic positions of the collaboration partners.

7.2.1 Servitization and Investment Preferences The market is diversified between companies looking at capital costs only and those considering total cost of ownership. Data centers looking to minimize investment costs will be more susceptible to cooling service options, while the other group will be more inclined to invest in local heat recovery. Given a different mindset, the acceptable price levels will also differ. The low interest rates have favored investments for those with capital. The service trend would probably be even more distinct with elevated interest rates.

7.2.2 Availability Each individual data center must dimension the investments to serve their availability ambitions, closely tied to the target client group. In the most ambitious data centers, district cooling can only serve as a complement, added on another redundant cooling solution. The district cooling can reduce the need for redundancy in the other systems, but never replace it fully, as it relies on the power grid. District heating can only be considered a complete solution for data centers without local backup power generation possibilities. Availability is derived from the probabilities of all risk factors in the system, meaning that redundancy is only part of the considerations to secure uptime. The district energy company seeking to establish a collaboration should reflect on what they can offer in terms of availability and total system reliability.

7.2.3 Temperature The temperature in a conventional data center is around 20 oC, and the tolerated temperature level seems to be successively increasing. The temperatures available for heat recovery are affected by the applied cooling technology and the location of heat recovery. The temperature and amount of heat possible to recover decreases with distance from the components producing the heat. From a heat recovery perspective, it would be preferable to collect the heat as close to the components as possible. However, this increases complexity in the system and cost, and is difficult to implement in an already established data center. These technologies are not yet relevant to most data centers, as conventional cooling technologies are sufficient for most actors and heat recovery is not considered core business. The heat pump is commonly supplied with temperatures around 20 oC, requiring significant elevation before supply to the district heating grid. Heat recovery via the district cooling grid does not have the same temperature requirements. As long as the heat is recovered at a temperature above average temperature in the return pipes of district cooling (usually around 16 oC), capacity in the system is increased. This temperature should not be hard to exceed for most data centers. Gathered empirics have also shown that certain data centers would prefer a higher supply temperature for cooling than 69 what is offered by the district cooling grid. Energy companies with district cooling could thus utilize the return pipes of district cooling for data center cooling, which entails new business opportunities. There are already similar solutions on the market, where supply and return temperature from district cooling is mixed before supplied to a data center. However, such solutions may not be of interest to all data centers, as required supply temperature is dependent on how modern the applied cooling technology is. The discussed temperature levels in this thesis should not be considered as exact values for the different technologies, as they are a result of qualitative analysis and interviews, and no measurements have been made. The temperature levels should instead be considered indications of the different temperature levels in a data center, suitable as a tool to help evaluate possible heat recovery solutions.

7.2.4 Power Density The values for power densities discussed in the gathered empirics are coherent with the values stated by previous literature and global surveys. For third-party actors, power densities are quite low compared to installed capacity, which limits the expected amount of heat with potential for extraction. The low values for power densities are due to a combination of reasons such as customers’ misunderstanding for their own capacity needs and security margins for increased capacity. Higher values for power density can be found in HPC clusters and other types of energy-intensive IT operations. An energy company seeking collaborations for heat recovery should be aware that, depending on operations, a data center claiming to have 3 MW installed capacity probably won’t provide the same amount of heat load. Managed hosting, i.e. when the data center chooses the servers and manage their capacity themselves, might provide slightly better values for power density compared to pure colocation operations, where customers place their servers in a data center. Power density is an important consideration when choosing a cooling solution. For third-party actors, the current power density values seem to be too low to incentivize investment in liquid cooling, and in some cases rack-cooling as well. Only energy-intensive IT operations have to apply rack-cooling or liquid cooling, as conventional cooling designs are not sufficient for such high power densities. As the values for power densities are not expected to increase significantly in the coming years, liquid cooling cannot be expected to become a conventional technology. If Norrenergi builds a data center for colocation purposes in their facilities, they should consider what type of IT-operations to host, as it would have a high impact on the potential amount of heat available for recovery. Additionally, Norrenergi would have the opportunity to optimize data center infrastructure for heat recovery. This could prove to be a project of academic interest.

7.3 Reliability and Validity In this section, we discuss the method and purpose fulfillment of the study.

7.3.1 Reliability Literature was collected using a multitude of search terms in both English and Swedish on industrial symbiosis, district heating, data centers, and heat recovery.

The methodology of using semi-structured interviews makes the study hard to reproduce due to several subjective factors. However, the line of thoughts in the analysis should be clear, as all interviews were recorded and transcribed. The responses were then thematically categorized and compiled, and are attached in Appendix C: Interviews with Data Center Industry Representatives. The broad range of interviewees gathers thoughts from different perspectives and increases credibility of our projections on the future market. We lack representation from in-house data centers and Internet providers, but interviewees from data center suppliers compensate for the potential shortcoming.

7.3.2 Validity The validity is a measure of purpose fulfillment and the study examining the phenomena set out to be studied (Blomkvist and Hallin, 2015). We have used the theories on industrial symbiosis to keep the study on track, when researching synergies, enabling factors and obstacles for the industry collaborations. From Chertow (2008) we used the three levels of exchange streams as a checklist for potential synergies and obstacles: 1. By-product reuse; 2. Utility/infrastructure sharing; 3. Joint provision of services. Each level has generated several points of interest, which we have followed up in the analysis. 7.3.2.1 Construct Validity Construct validity is a test to reduce the bias risks in the study (Yin, 2014). The theories on industrial symbiosis are taken from the work of M.R. Chertow (2000; 2008), who is the author of the concept. Her theories are backed up by other case studies on industrial symbiosis and eco-industrial parks, which we also cover in our literature review. For the data center and district energy markets, we have used multiple sources of evidence to identify converging phenomena. The subjectivity in the answers from semi-structured interviews is compensated by literature and a diversity in interviewee selection. The white papers used are only to see as complementary evidence, as they are usually written in a promotional perspective. We have also asked our interviewees to review the outcomes of the interviews, ensuring the quality of the interview syntheses. 7.3.2.2 External Validity This is a single case study, with base on the specific conditions for a district and cooling company with no power production. The recommendations for value propositions are generalizable, to the extent that other energy companies recognize the paradigm of Norrenergi, since their values represent the whole energy company side of the collaborations with data centers. Some argumentation is only applicable for energy companies with similar product and service offer, location, and fuel mix, while some is generalizable to all energy companies, wishing to integrate data centers for recovery of residual heat. Competition with substitutes on the district heating and cooling market will be present for all energy companies, and the compensation dilemma will be the same for the whole industry. The research of trends in the data center industry is focused on the Swedish market, and might not be applicable in other regions.

7.4 Suggestions for Future Studies The limitations in time and scope have left some stones unturned for future research. Complementary research subjects for this study are a quantitative analysis of data center and energy company collaborations and business case analyses of already initiated projects.

7.4.1 Quantitative Analysis This thesis has applied a qualitative analysis in order to gain a rich and deep understanding of the complexities surrounding heat recovery from data centers. Based on the conclusions made in this thesis, future studies could focus on collecting quantitative data regarding data centers preferences for heat recovery. It could be of interest to ask data centers to rate different types of value propositions from energy companies, or ask them to rank different factors, to help determine what type of value propositions that would appeal to the majority of data centers.

7.4.2 Business Case This thesis has presented different factors and business opportunities which are important to consider for an energy company when approaching a heat recovery project with a data center. In order to further evaluate different configurations for heat recovery and potential business models, concrete business cases for each configuration is needed. This thesis has performed no economic calculations to evaluate possible solutions, and thus a suggestion for further studies is to formulate business cases for the desired configurations. The business cases should contain would-be partners such as a specific data center with specified characteristics, and a possible data center supplier. Secondly, we suggest to perform case studies on established collaborations to gain deeper knowledge on the factors which made those collaborations successful or not. During our study, we have interviewed data centers who are part of heat recovery collaborations. Their experiences are thus reflected in our results. However, a deeper case studies analysis would gain more insights.

8 Conclusions The purpose of the study was to explore the potential for district energy companies to integrate data centers into their system as a source for residual heat and determine possible synergies, enabling factors and obstacles for such collaborations. We have fulfilled the purpose by applying a holistic approach, where trends and requirements in data center operations have been compared to trends and requirements from district energy companies. This chapter presents the general conclusions made from our study. Collaborations between data centers and energy companies offer synergies related to each level in the framework for industrial symbiosis: By-product reuse; Utility/infrastructure sharing; and joint provision of services. The synergy characteristics provide good prospects for mutual benefits from collaborations.

The data center industry is a diversified industry, where data centers can differ vastly from each other in regards to customers, availability requirements and business models. Energy companies seeking collaboration need to formulate products and service offers, which allow for flexibility towards local conditions. Some important factors are heat quality prospects, proximity to district energy systems, and issues related to control and availability.

As data center operation becomes increasingly focused on limiting capital expenditures, district energy companies should examine the possibilities to expand the interface. District energy companies should evaluate the potential of taking on operations responsibility and ownership of the local infrastructure, necessary for heat recovery.

To avoid losing heating and cooling customers to local solutions, energy companies should increase accessibility to district cooling, enabling efficient solutions with low local investment costs.

Heat recovery is not part of the core business of data centers, which impedes willingness to instigate collaboration. However, the environmental trend can help motivate projects for heat recovery, by encouraging eco-friendly solutions.

Heat quality is not expected to increase significantly short term, but varies between businesses. Only special data center segments, e.g. HPC, will be inclined to work with higher temperatures and heat densities.

Dedicated roles are a success factor for driving the development projects and are particularly important when integrating different industries. Collaboration and partnership are key notions in all discussions on heat recovery projects and implies a shift from a pure supplier-customer relationship to a collaboration, with common goals and increased transparency.

9 References

9.1 Scientific Articles and Books Barroso, L.A., Clidaras, J. and Hölzle, U., 2013. The datacenter as a computer: An introduction to the design of warehouse-scale machines. Synthesis lectures on computer architecture, 8(3), pp.1-154. Behera, S. K., Kim J. H., Lee S. Y., Suh S. and Park S.H. 2012 Evolution of ‘designed’industrial symbiosis networks in the Ulsan Eco-industrial Park: ‘research and development into business’ as the enabling framework.,Journal of Cleaner Production 29, 103-112. Berk, J. and DeMarzo, P. 2014. Corporate Finance. Harlow: Pearson Education Limited, p. 234-241. Björnsdotter, A., 2012. Återvinning av industriell restvärme som värdeskapande process: En fallstudie på SSAB EMEA i Borlänge. Master thesis. Royal Institute of Technology, School of Industrial Engineering and management. Blomkvist, P. and Hallin, A., 2015. Method for engineering students - Degree projects using the 4-phase Model. Lund, Sweden: Studentlitteratur AB, pp. 66 Brecheisen, S. and From, R., 2013. Marknadsundersökning inom datacenterbranschen: Undersökning av tekniska inköp, med fokus på kyllösningar. Master thesis. Uppsala Universitet, Industrial Management and Innovation. Chertow, M.R., 2000. Industrial symbiosis: literature and taxonomy. Annual review of energy and the environment, 25(1), pp.313-337. Chertow, M.R., 2007. “Uncovering” industrial symbiosis. Journal of Industrial Ecology, 11(1), pp.11-30. Chertow, M.R., Ashton, W.S. and Espinosa, J.C., 2008. Industrial symbiosis in Puerto Rico: Environmentally related agglomeration economies. Regional studies, 42(10), pp.1299-1312. Cooper, D. R. and Schindler, P.S., 2014. Business Research Methods. 12th edition. New York, The United States of America: The McGraw-Hill Companies, pp.165 Davies, G. F., G. G. Maidment, and R. M. Tozer.2015. Using data centres for combined heating and cooling: An investigation for London. Applied Thermal Engineering 94 (2016): 296-304. Ebrahimi, K., Jones, G.F. and Fleischer, A.S. 2013. A review of data center cooling technology, operating conditions and the corresponding low-grade waste heat recovery opportunities. Renewable and Sustainable Energy Reviews, 31, pp.622-638. El Azziz, C. and Izadi, R., 2012. Green Room: A Giant Leap in Development of Green Datacenters. Master Thesis. Royal Institute of Technology, School of Industrial Engineering and management. Greijer, A., 2010. Energieffektivitet i datahallar – en jämförelsemetod och deklaration för datahallar med avseende på energieffektivitet & miljöbelastning. Master Thesis. Royal Institute of Technology, Industrial Engineering and Management. 74

Ilic, D.D. and Trygg, L., 2014. Economic and environmental benefits of converting industrial processes to district heating. Energy Conversion and Management, 87, pp.305- 317. Kindström, D., Ottosson, M., Thollander, P. and Kienzler, M., 2015. Energitjänster i energibolag: ett ökat värdeskapande med kunden i fokus. Lindahl, M., Sundin, E. and Sakao, T., 2014. Environmental and economic benefits of Integrated Product Service Offerings quantified with real business cases. Journal of cleaner production, 64, pp.288-296. Little, A.B. and Garimella, S., 2011. Comparative assessment of alternative cycles for waste heat recovery and upgrade. Energy, 36(7), pp.4492-4504. Lombardi, D.R. and Laybourn, P., 2012. Redefining industrial symbiosis. Journal of Industrial Ecology, 16(1), pp.28-37. Lombardi, D.R., Lyons, D., Shi, H. and Agarwal, A., 2012. Industrial symbiosis. Journal of Industrial Ecology, 16(1), pp.2-7. Magnusson, D., 2012. Swedish district heating—A system in stagnation: Current and future trends in the district heating sector. Energy Policy 48 (2012): 449-459. McEwen, O., 2015. Potentiellt utnyttjande av spillvärme för fjärrvärmenätet i Västerås. Morandin, M., Hackl, R. and Harvey, S., (2014). Economic feasibility of district heating delivery from industrial excess heat: A case study of a Swedish petrochemical cluster. Energy, 65, pp.209-220. Olsson, L., Wetterlund, E. and Söderström, M., 2015. Assessing the climate impact of district heating systems with combined heat and power production and industrial excess heat. Resources, Conservation and Recycling, 96, pp.31-39. Pakbaznia E., Pedram M., 2016. Minimizing Data Center Cooling and Server Power Costs. (2016-11-30) Persson, U. and Werner, S., 2011. Heat distribution and the future competitiveness of district heating. Applied Energy, 88(3), pp.568-576. Päivärinne, S. and Lindahl, M., 2015. Exploratory study of combining Integrated Product and Services Offerings with Industrial Symbiosis in order to improve Excess Heat utilization. Procedia CIRP, 30, pp.167-172. Srivastava, P. K., Kulshreshtha, K., Mohanty, C. S., Pushpangadan, P., & Singh, A. 2005. Stakeholder-based SWOT analysis for successful municipal solid waste management in Lucknow, India. Waste management, 25(5), 531-537. Söderman, Jarmo, and Frank Pettersson. 2006. Structural and operational optimisation of distributed energy systems. Applied Thermal Engineering 26.13: 1400-1408. Tsvetkova, A. and Gustafsson M. 2012. Business models for industrial ecosystems: a modular approach. Journal of Cleaner Production 29, 246-254. Valentin, E. K. 2001. SWOT analysis from a resource-based view. Journal of Marketing theory and Practice, 9(2), 54-69. Wang, L. and Khan, S.U., 2013. Review of performance metrics for green data centers: a taxonomy study. The journal of supercomputing, 63(3), pp.639-656.

Yin, R.K., 2014. Case study research: Design and methods. Sage publications.

9.2 Other Written References 42U.com, 2009. Hot Aisle/Cold Aisle Layout for Data Center Racks. [online]. Available at: http://www.42u.com/cooling/hot-aisle-cold-aisle.htm [2017-05-22] Aboutcolocation, 2017. Colocation Vs Managed Hosting.[online]. Available at: http://aboutcolocation.info/colocation-vs-managed-hosting/ [2017-02-09] Andersson, M. and Bergqvist, A. 2016. Lagrådsremiss – Vissa frågor på elskatteområdet. Stockholm: Finansdepartementet, p.6-7. Available at: https://www.energiforetagen.se/globalassets/energiforetagen/nyheter/2016/juni/ vissa-fragor-pa-elskatteomradet.pdf?v=nonce-8c0e2402-9d12-45ea-972f- 8f14da589dc7 [2017-05-10] Arnell J., Bolin L., Holmgren K., Staffas L., Adolfsson I., Lindblad M., 2012. Förutsättningar för ökad nytta av restvärme, Svenska Miljöinstitutet (Report B2077), Fjärrsyn (Report 2012:14). Available at: http://www.svenskfjarrvarme.se/Fjarrsyn/Forskning--Resultat/Ny- kunskapresultat/Rapporter/Omvarld/Forutsattningar-for-okad-nytta-av- restvame-/ [2016-11-15] ASHRAE Technical Committee, 2016. Data Center Power Equipment Thermal Guidelines and Best Practices. Available at: https://tc0909.ashraetcs.org/documents/ASHRAE_TC0909_Power_White_Pape r_22_June_2016_REVISED.pdf [2017-05-10] Bahnhof, n.d. Thule produktblad. Available at: https://www.bahnhof.se/filestorage/userfiles/file/Thule%20produktblad.pdf [2017-02-10] BCG, 2016. Capturing the Data Center Opportunity. Available at: http://www.business- sweden.se/contentassets/cd7d2c2584d64e8694e92ec1f6408069/bcg-capturing- the-data-center-opportunity-june-2016.pdf [2016-12-03] Cook, G., Lee, J., Tsai, T., Kong, A., Deans, J., Johnson, B., Jardim, E., 2017. Clicking Clean: Who Is Winning The Race To Build A Green Internet?. Washington D.C.: Greenpeace, p. 5. Available at: https://secured- static.greenpeace.org/austria/Global/austria/dokumente/Clicking%20Clean%202 017.pdf [2017-05-17] Coromatic, 2015. Inside the shadow of the cloud. Rev#1. Available at: https://coromatic.se/wp-content/uploads/2015/11/Coromatic-In-the-shadow-of- the-cloud-2015.pdf [2017-02-09] Coromatic, 2017. As a Service [online]. Available at: https://coromatic.se/tjaenster/as-a- service/ [2017-03-09] Cushman & Wakefield, 2016. Data Center Risk Index – Informing global location strategies in a digital world expanding at a phenomenal pace 2016, p. 2-9. Available at: http://www.cushmanwakefield.com/en/research-and-insight/2016/data-centre- risk-index-2016/ [2017-03-13]

DCD Intelligence, 2015. Colocation Provider Investment: Co-opting the Cloud For Future Growth. Available at: www.datacenterdynamics.com/download?ac=13067 [2017-05-08] DCD Intelligence, 2015b. Anixter, global technology briefing – Thermal Efficiency Best Practices. Available at: www.datacenterdynamics.com/download?ac=14080 [2017-05-11] DCD Intelligence, 2016. The 2016 Data Center Trends Survey. [online]. Available at: http://www.fdcf.fi/wp-content/uploads/2016/09/Key-Results-from-the-2016- Global-Data-Center-Survey-Respondents.pdf [2017-05-11] EcoDataCenter, 2017. Making the Most of Energy Used. Available at: https://ecodatacenter.se/heat-reuse/ [2017-05-10] EcoDataCenter b., 2017. The World’s First Climate Positive Data Center. Available at: https://ecodatacenter.se/#new-page [2017-05-10] Elementica Data Center Construction AB, 2015. Välkommen att investera i framtidens datacenter. ElementicaMemorandum. Available at: http://investor.elementica.se/ElementicaMemorandum.pdf [2017-02-10] Elementica, 2017. Overview. Available at: http://www.elementica.se/ [2017-05-07] Emerson Network Power, 2011. Understanding the Cost of Data Center Downtime: An analysis of the Financial Impact on Infrastructure Vulnerability. Available at: https://www.scribd.com/document/92299048/1-14062-Understanding-the-Cost- of-Data-Center-Downtime-ENP Energimarknadsinspektionen, 2013. Kartläggning av marknaden för fjärrkyla. EiR2013:18. Eskilstuna, Sweden: Elanders Sverige AB. Available at: http://ei.se/Documents/Publikationer/rapporter_och_pm/Rapporter%202013/EI_ R2013_18.pdf [2016-12-05] Fastighetstidningen, 2016. Datorer värmer studenter. Fastighetstidningen (6) s.55-56. Available at: http://paper.opoint.com/?id_site=87783&id_article=1438&code=403 [2016-11- 15] FEV, 2017. Datacenter byggs i Falun. Available at: http://www.fev.se/privat/aktuellt/datacenter-byggs-i-falun/ [2017-05-10] Fortum Värme, 2017. White Paper on Data Center Cooling and Heat Recovery. Available at: https://stockholmdataparks.com/resources/ [2017-05-02] Greenberg, A., Hamilton, J., Maltz, D.A. and Patel, P., 2009. The Cost of a Cloud: Research Problems in Data Center Networks, Microsoft Research, Redmond. Available at: http://research.microsoft.com/en-us/um/people/dmaltz/papers/dc-costs-ccr- editorial.pdf [2016-11-15] Hewlett Packard, 2017. Hybrid IT: Bringing together the best of IT infrastructures. Available at: https://www.hpe.com/h20195/v2/getpdf.aspx/a00002363enw.pdf?ver=5.0 [2017-05-13]

IDG Connect, 2016. IDG and DigiPlex survey Data centre trends 2016. Available at: http://www.digiplex.com/locations/trends [2017-02-13] Industry Perspectives, 2015. The Importance of Scalability and Cost of Data Center. Data Center Knowledge. Available at: http://www.datacenterknowledge.com/archives/2015/08/05/importance- scalability-cost-data-center-solutions/ [2017-05-11] Johansson, L., 2016. Bahnhof Sänkt skatt för datahallar från den 1 januari 2017. Placera.. [online]. Available at: https://www.avanza.se/placera/telegram/2016/11/25/bahnhof-sankt-skatt-for- datahallar-fran-den-1-januari-2017-bahnhof-valkomnar-beslutet.html [2017-03- 06] Kleyman, B. 2016. Data Center Liquid Cooling – How New Cooling Technologies are Driving the Compute and Cloud Revolution. [online]. Data Center Frontier. Available at: http://datacenterfrontier.com/liquid-cooling-solution-data-center- cooling/ [2017-05-11] Miller, R., 2015. Data Centers on the Edge: Streamin and IoT Reshape the Network. Data Center Frontier, [online]. Available at: http://datacenterfrontier.com/edge-datacenters/ [2017-02-13] Miller, R., 2017. The Year Ahead: Edge Computing Creates Opportunity in 2017. Data Center Frontier, [online]. Available at: http://datacenterfrontier.com/the-year- ahead-edge-computing-creates-opportunity-in-2017/ [2017-02-13] Mortenson, 2014. Insights Into What’s Next – Trends in Data Centers. [online]. Available at: http://www.mortenson.c om/~/media/files/thought%20leadership/data-center- trends-mortenson-construction.ashx [2017-05-07] Node Pole, 2016. Swedish Parliament approves lowered tax rates for data centers from January 1st, 2017. [online] Available at: http://thenodepole.com/2016/11/24/swedish-parliament-approves-lowered-tax- rates-data-centers-january-1st-2017/ [2017-03-06] Norin, A. 2016.FaluEnergi & Vatten AB och Falu kommun ändrar planerna för datahallarna. Södra Dalarnes Tidning. Available at: http://www.dt.se/dalarna/falun/faluenergi-vatten-ab-och-falu-kommun-andrar-planerna-for-datahallarna [2017-05- 10] Norrenergi, 2015. Stort steg mot långsiktigt produktionssamarbete om fjärrvärmen mellan Sundbyberg, Solna och Stockholm. Available at: http://www.norrenergi.se/press-och-nyheter/nyheter/langsiktigt- produktionssamarbete/ [2017-05-01] Norrenergi, 2016. Våra nät gör det möjligt. Available at: http://www.norrenergi.se/fjarrvarme- fjarrkyla/vara-nat/ [2016-11-15] Norrenergi, 2016b. Hur fungerar vår fjärrvärme?. Available at: http://www.norrenergi.se/fjarrvarme-fjarrkyla/hur-fungerar-det-hos- oss/norrenergis-fjarrvarme/ [2016-11-15] Norrenergi, 2016c. Samråd om utvecklingen av Solnaverket och bostäder i sydvästra Huvudsta. Available at: http://www.norrenergi.se/press-och-

nyheter/nyheter/samrad-om-utvecklingen-av-solnaverket-och-bostader-i- sydvastra-h/ [2016-11-27] Norrenergi, 2017. För ett samhälle i balans. Available at: http://www.norrenergi.se/norrenergi-dig/om-oss/ [2017-05-22] Norrenergi, 2017b. Available at: http://www.norrenergi.se/media/filer_public/5b/9d/5b9de924-b824-4947-8b26- c9f7d3c1bb1d/preliminar_miljoprestanda_norrenergi_2016.pdf [2017-05-22] Norrenergi, 2017c. Norrenergi återvinner överskottsvärme från datacenter i Solna. Available at: http://www.norrenergi.se/press-och-nyheter/nyheter/norrenergi-atervinner- overskottsvarme-fran-datacenter-i-solna/ [2017-05-01] Open District Heating, 2017. A New Market for Recovered Energy. Available at: https://www.opendistrictheating.com/a-new-market-for-recovered-energy/ [2017-05-10] PDC, n.d. PDC Center for High Performance Computing, informative poster. Stockholm. Rasmussen, N, 2011. Determining Total Cost of Ownership for Data Center and Network Room Infrastructure. Schneider Electric – Data Center Science Center, White Paper 6 Rev 4. Rasmussen, N, 2011b. Calculating Total Power Requirements for Data Centers. Schneider Electric – Data Center Science Center, White Paper 3 Rev 1. Rasmussen, N, 2012. Avoiding Costs from Oversizing Data Center and Network Room Infrastructure. Schneider Electric – Data Center Science Center, White Paper 37 Rev 7. Reichenberg, L., Andersson, E., Åsblad, A., 2011. Drivkrafter för energisamarbeten – möjligheter och hinder. CIT Industriell Energi AB. Rydén B., Sköldberg H., Stridsman D., Göransson A., Sahlin T., Sandoff A., Williamsson J., Hansson N., Holmberg U., Gunnarsson A. (2013). Fjärrvärmens affärsmodeller. Fjärrsyn rapport 2013:17. Available at: http://www.svenskfjarrvarme.se/Fjarrsyn/Forskning--Resultat/Ny- kunskapresultat/Rapporter/Marknad/Fjarrvarmens-affarsmodeller-/ [2016-11- 29] Serverlift, 2012. Data Center as a Service (DCaaS). Available at: http://serverlift.com/tech- lift/data-center-as-a-service-dcaas/ [2017-02-09] Smolaks, M., 2017. Interxion set to build €29 million facility with Stockholm Data Parks. DatacenterDynamics. Available at: http://www.datacenterdynamics.com/content-tracks/power-cooling/interxion- set-to-build-29-million-facility-with-stockholm-data-parks/98030.article [2017- 05-02] Stockholm Data Parks, 2017. Available at: https://stockholmdataparks.com/ [2017-05-02] Stockholm Data Parks, 2017b. Stockholm Data Parks – Green Computing Redefined. Available at: https://stockholmdataparks.com/resources/ [2017-05-02]

Svensk Fjärrvärme, 2016. Tillförd energi. Available at: http://www.svenskfjarrvarme.se/Statistik--Pris/Fjarrvarme/Energitillforsel/ [2016-11-15] The Swedish Energy Agency, 2013. EU:s system för handel med utsläppsrätter. Bromma: Arkitektkopia, p. 7-13. Available at: https://energimyndigheten.a- w2m.se/Home.mvc?ResourceId=2690 [2017-03-06] The Swedish Energy Agency, 2016. Risken för avbrott i fjärrvärme – Utredning om fjärrvärmeföretagens ekonomiska ställning samt deras förmåga att förebygga och åtgärda avbrott. Eskilstuna: Energimyndigheten, p. 6;11-19; 35-43; 53-57. Available at: https://energimyndigheten.a- w2m.se/FolderContents.mvc/Download?ResourceId=5540 [2017-05-09] The Price Dialogue, 2017. Prisdialogen Mellan kunder och fjärrvärmeföretag. [online]. Available at: http://www.prisdialogen.se/ [2017-03-07] The Swedish Government, 2016. Förändringar på skatteområdet i budgetpropositionen för 2017. Stockholm. Available at: http://www.regeringen.se/artiklar/2016/09/forandringar-pa-skatteomradet-i- budgetpropositionen-for-2017/ [2016-11-15] The Swedish Government, 2017. Miljö- och energiskattepaket inför höstbudgeten. [online]. Available at: http://www.regeringen.se/pressmeddelanden/2017/03/miljo--och- energiskattepaket-infor-hostbudgeten/ [2017-05-09] Uptime Institute, 2009. Data Center Site Infrastructure Tier Standard: Topology. Available at: https://uptimeinstitute.com/research-publications/asset/tier-standard-topology [2017-02-08] Wendel, J., 2016. Finanschef sålde alla aktier före negativt besked. Dagens Industri, [online], 2016-09-15. Available at: http://www.di.se/nyheter/finanschef-salde-alla-aktier- fore-negativt-besked/ [2017-05-07] Öppen Fjärrvärme, 2017. Om oss. Available at: https://www.oppenfjarrvarme.se/om-oss/ [2017-05-10]

9.3 Personal Communication Dahlgren, J., March 2017. Interview. Edén, T., February 2017. Interview. Fridström, M., March 2017. Interview. Gierow, M., March 2017. Interview. Granström, L., March 2017. Interview. Havtun, H. February 2017. Interview. Lindqvist, M., March 2017. Interview. Lundquist, J., March 2017. Interview. Nilsson, P., February 2017. Interview. Olrin, L., 2017. E-mail communication.

Svensson, G., March 2017. Interview. Stymne, S., December 2016. Introduction interview. Stymne, S., 2017. Personal communication. Stymne, S., March 2017. Interview.

9.4 DI Industry Fair, February 22, 2017. Stockholm. Bank, P., 2017. Interxion Nordic – The interconnected future and the Nordic market. Bäckström, P., 2017. Modular Data Center – 3 x advantages: fast, flexible and efficient. Eckhoff, G.M., 2017. Paneldiskussion: Affärsmässighet hos framtidens datacenter – vilka faktorer blir avgörande i skuggan av IT-jättarna? Egnell, G., 2017. Paneldiskussion: Affärsmässighet hos framtidens datacenter – vilka faktorer blir avgörande i skuggan av IT-jättarna? Graf, A., 2017. Move Data Not Power – A lesson in googlenomics for Nordic businesses. Lindqvist, M. and Hedman, J., 2017. Energieffektiva lösningar i framtidens datacenter. Lindström, E. 2017. Datacenter. Sokolnicki, T., 2017. Data centers by Sweden. Svanberg, C., 2017. Sverige och Norden har manga fina förutsättningar för att bli ett nav för nya datacenteretableringar – men hur säkerställs förutsättningarna över tid? Vyncke, F., 2017. Paneldiskussion: Affärsmässighet hos framtidens datacenter – vilka faktorer blir avgörande i skuggan av IT-jättarna?

Appendix A: Interviews with Norrenergi Representatives This Appendix is a bullet point presentation of interview responses, thematically categorized and compiled. Each statement is followed by acronyms of the author, in parenthesis. The acronyms represent the following interviewees: TE – Ted Edén SS – Staffan Stymne

A.1 Interview Questions (SS)

A.1.1 General How does Norrenergi categorize their customers?

A.1.2 Economy What investment requirements exist? What are the primary cost drivers for your business? What are the primary risks for your operations?

A.1.3 Strategy What do you mean by cyclic business approach? How do you plan to grow your business?

A.1.4 Servitization How would you define energy services? How do you work with energy services?

A.1.5 Pricing What is your pricing currently based on? What do you think about the price development?

A.1.6 District Cooling How important is the district cooling business related to the district heating business? What do you think about the market development for district cooling? What does it mean for you that temperatures in data centers are increasing?

A.1.7 Heat Recovery What possibilities do you see in collaborations with data centers? What difficulties do you see in collaborations with data centers? How do you consider making an investment beforehand if necessary? What demands would you have on a data center to start a collaboration?

Who should own a local heat pump? What do you think about commitments for heat deliveries? What obligations should the different parties have? What do you think about collaborations with other external organizations and companies?

A.1.8 Operations How reliable are your operations? How does your operations organization work?

A.1.9 Miscellaneous How does the Price Dialogue work?

A.2 Interview questions (TE)

A.2.1 Technology What information is necessary to secure quality of deliveries? What requirements are there regarding temperatures in the grid? How do you work with distribution losses? What does the load curve look like? - Short-term, long-term - How are changes managed? How does production planning work? How are deliveries of residual heat to the grid preferred? - Supply line vs. return line

A.2.2 Economy How are costs divided for new connections? Where is the boundary between Norrenergi’s responsibility and the customer’s responsibility?

A.2.3 Collaborations What is Norrenergi’s responsibility towards the customer in a heat recovery collaboration?

A.3 Interview Responses

A.3.1 Market Development • Historically, we have been focused on growing in volume in terms of energy deliveries, only counting the MWh. Now, we see a completely different development, with the number of customers increasing dramatically, while the energy volumes may not grow at all. Today, energy efficient properties are enforced politically. The combination of large customer influx and the general development in energy demand for each customer, makes it hard to assess the development for the total energy demand in the system. Our earlier definition of a normal customer has been revised, in the light of new combinations of heating and cooling solutions. We must be flexible in our business approach and adapt our business model to every new situation. (SS) • Probably, we have to get used to only being a part of the solution for some customers, where they buy our services during certain periods and in varying volumes. (SS) • Both district heating and district cooling are important areas for our development. The district heating is 10 to 15 times bigger, and definitely our main business. But the district cooling is a rather young product, some 30 years behind the district heating. I think the relation in volumes for delivered heating and cooling will change over time, with an increasing cooling share. The combination of heating and cooling is interesting, as one is the by-product of the other. That gives leverage for both businesses, if done right. (SS) • The customer segmentation that takes place, is based on the activities in the specific property. It may be housing, commercial activities, or industries. The most common customers in the housing segment are housing cooperatives. Private customers are a separate category, mainly consisting of detached houses. (SS)

A.3.2 Cyclic Business Approach • We have chosen the term cyclic business approach, because it is closely related to circular economy. We have basically moved from a linear business approach; incoming fuel, distribution, and deliveries. Our business model is to deliver the right quality to the right place, to the lowest rate. Now, we are using a system approach, and our role has shifted to becoming responsible for the system balance, while the production can come from several parties. The system is a network of output and input points for energy, and the end customer, which is included in the system, may also be a part of the production. (SS) • Our new role will in addition to the role as supplier, be to manage changes in load and production from suppliers of residual heat. Our current base production consists of heat pumps and a supply agreement with neighboring district heating companies complemented with peak load production. The idea is that we receive the recovered heat as a base and then regulate it with the facilities we have. (TE)

A.3.3 Servitization • An energy service is everything we provide for our customer outside of the core delivery of heat and cool. It may be a service agreement for and supervision of district heating centrals, owned by the customer. It may also be energy statistics. (SS)

iii

• We haven’t decided if we should own heat pumps at our clients’ sites, meaning that we take the investment instead of the client. Until now, it hasn’t been relevant, and there are still several questions to solve, prior to such a solution. However, I don’t want to rule it out completely. (SS) • How much to servitize, depends on how far we should approach the businesses of our clients, and how we chose to position ourselves. If large enough, we could consider data centers as alternative production facilities. I don’t think we can go all the way to becoming data center owners, but I think we could go as far as providing spacing and infrastructure for power, heating and cooling. In that case, we would get the most synergies with our core business, since we already have the competence, control rooms and around the clock operation staff. (SS) • I can’t rule out a layout, where we rent out capacity on our heat pumps, but we have no such constructions today. We would only consider it if it implied synergy effects for both parties and would have to be evaluated together with the potential partner. We don’t want to limit ourselves on beforehand, but be open to new possibilities. (SS) • The limit of Norrenergi’s area of responsibility is often drawn at the border of the property. Norrenergi builds pipes into the property, and thereafter the customer takes responsibility for the construction. The same goes for maintenance and operations. Normally Norrenergi charges a fixed cost for the connection. To broaden the area of responsibility for Norrenergi to include maintenance and operation of for example the customer’s cooling technology is foremost an economic issue, but this might change due to customer demand. Additionally, there are already established companies working with this kind off services and I find it difficult to believe that we, today, would be more competitive than them. Currently, Norrenergi’s staff in production and operations haven’t that much experience from operating cooling machines in properties yet. (TE) • Today it is not possible to completely replace data center suppliers since data centers always want redundancy in their systems for cooling. (TE) • So far, the data centers have operated the heat pumps, themselves. As we have some experience in heat pump maintenance, we could expand our service offer in the future, to customers without that knowledge. (TE)

A.3.4 Investments • We do a lot of different projects and the investment horizon differs with the nature of the projects. Projects regarding working environment usually have a shorter payback time, which allows for short-term planning in that area. Production renewal projects usually have a 5 to 8-year realization period, and those projects are long-term. First, we have a decision process in regards to strategy, fuels etc. After the decision process, the construction of a new boiler may take several years, excluding different permit procedures. (SS) • The main cost drivers are commodities and supplies. Fuel, including electricity price, is the single most important cost, and it drives the investment strategy of the company. (SS) • We utilize capital budgeting. There are loads of factors to consider. City locations mean limitation to certain fuels and in our area, we can’t invest in incineration facilities. (SS)

• One of our toughest investment tasks is the dimensioning of piping system extensions. All sort of over-dimensioning is beyond budgeting for the specific investment, and would be made in speculation. In hindsight, many dimensions in historic extensions could have been made larger, but it is hard to manage uncertain investments. We are usually not comfortable in taking such risks. We can consider building in advance, e.g. over-dimensioning, if we have identified a potential in an area where we are already carrying out extension projects. (SS) • Roughly, investment costs for the pipes to the customer’s facility are linear. If only half of the power capacity is used, construction was probably twice as expensive as necessary. In general, it´s important that the customer know the size of the facility from the start, to be able to give them competitive offers. (TE) • Sensitivity analyses are made for investment costs to assess the risk. (TE)

A.3.5 Fuel Mix and Pricing • We are relatively dependent on biofuel prices, but it is not our base production. That means that biofuel prices only have a moderate effect on our district heating prices. (SS) • Except for our big heat pumps for base production, we have a distribution agreement with another energy company, making us feel comfortable with our price development on the heat market. (SS) • If the electricity price increases, district heating will become more cost-efficient in relation to local heat pumps, as they don’t have a mix of different fuels. One of the strengths of district heating is a fuel flexibility over time. Also, we are connected to the whole Stockholm system, including a lot of combined power and heating, where the energy companies hedge each other’s risks. (SS) • On the district heating side, we are part of the “Price Dialogue”, in which we have declared our pricing principles. Our pricing is cost based, with competition based elements. We are primarily comparing our prices to other district heating companies. (SS) • It is hard to compare ourselves to heat pumps, as we value capital differently. This makes it hard to evaluate the alternative costs, leading to problems with assessing our own competitiveness in some cases. (SS) • I believe we are competitive in regards to our solutions, as we are a good alternative both economically and environmentally. (SS)

A.3.6 Load Management and Production Planning • Recovered heat is prioritized as base production in the production planning, since those facilities can’t be disconnected and is expected to have the lowest production cost. Consensus is that residual heat should be utilized if technologically possible. (TE) • With a higher share of residual heat from data centers, we will need to know more about their heat profile, to be able to plan for efficient use (TE) • The daily planning is the most important. You never plan for shorter time intervals than one hour. For shorter intervals, there is always some part of the system which manages the fluctuations, for example a boiler or an accumulator. We can manage

quite large fluctuations in production. We control the differential pressure in the system and the distribution pumps according to demand. (TE) • Burning municipal waste basically entails negative costs during the summer, which means it is prioritized before heat pumps. When heat pumps are needed to produce cooling, they are exempted. It is however unlikely that the heat pumps would be prioritized because of cooling production, in the long run as there normally are both cooling machines and accumulators for free cooling. (TE)

A.3.7 Risks • One risk is the development of the fuel prices and possible tax changes. Regulations change over time, and the preconditions can change several times during an investment cycle. (SS) • We gradually become more dependent on IT, and have therefore focused more on IT security and intrusion threats. (SS) • Other risks are connected to the surrounding world. The water treatment plant, from which we get a lot of residual heat, is closing in a couple of years. Even though it was predictable, it has a major effect on our company. We have made large investments in heat pumps, which rely on residual heat, and now we must adjust to the new conditions in our area. (SS) • Future demand and competition with substitutes are the constant challenges for our business. (TE)

A.3.8 Managing District Energy Systems • With a differentiation of cooling demand, we don’t have to focus on the two products heat and cool. Instead, we can sell four different energy qualities as a service: cool supply flow of 6 °C, cool return flow of 15 to 16 °C, heat return flow of 45 °C, and heat supply flow of 70 to 80 °C. We would be less limited if we could start working with different energy qualities, instead of only having two products. (SS) • We continuously work with decreasing the return temperatures in the heating system, and increasing the return temperatures in the cooling system. Every degree Celsius of improvement has a value, and that could be incorporated into our contracts. Also, higher temperature differences increase the capacity of the distribution. Especially in the current cooling system. Today, we have a temperature difference of about 10 °C in the district cooling system. (SS) • We lack data on the standard of the properties in some parts of our system. When investigating the possibilities of lowering the temperature in the pipes, we make a local inventory. We do that anyways, to ensure enough flows in the area. (SS) • The quality of delivery to both district heating and district cooling customers is dependent on two things: temperature and differential pressure at the specific location. The temperature effects the sizing of the customer’s facility and system. The temperature is related to the present temperature demands and differential pressure is related to the sizing of valves. A third factor is total pressure, which is related to solidity and not quality of delivery. We design district heating for 16 bars and 120 °C in total. The customer’s facility must be constructed accordingly. (TE)

• Oversizing is common. An interval for differential pressure is chosen between 1 and 8 bars. Depending on location the interval could be smaller; between 2 and 4 bar or 2 and 5. (TE) • Differential pressure varies between the inner parts of the grid and the outer parts. IT could decrease from 7-8 bars in the inner parts to 2 bars further out, before it is pumped up again. The temperature decreases with a couple of degrees between the inner parts and the outer parts; some 3 °C during winter and close to 5 °C during summer due to decreased flows. (TE) • The lowest temperature we can deliver to a customer is 65 °C. When the outside temperature decreases to 2 °C we increase supply temperature linearly up to 100 °C at -18/-19 °C outside temperature. TE) • For district cooling, we demand 10 °C temperature difference between the supply and the return flows. For district heating we have no such demand, due to its diverse applications. Temperatures vary depending on building characteristics. Modern buildings can manage lower temperatures in their heating systems. (TE) • The customers design their facilities for input of 65 to 100 °C from district heating. We operate the system on a bit higher temperatures, partly because we want to keep the flows down and partly because our system is a bit tight. By increasing temperature, the power increase can be achieved at the same mass flow. Maximum flow is at 2 °C outside temperature. (TE)

A.3.9 Data Center Heat Input • In regards to district cooling, higher temperatures in data centers are positive for us, as higher return temperatures are favored by our heat pumps. If a data center has equipment to handle large temperature differences, they could provide good return temperatures from the district cool supply flow. However, if we could cool data centers with the return flow in the cooling system, we would utilize our investments optimally, increasing the capacity of the system. An increase in temperature from 15 to 25 °C, would almost double the capacity in our investments. (SS) • If we have both district heating and cooling in proximity to a data center, we could benefit even more from our big system. Depending on the temperatures in the different systems, and on the heat demand, we could choose on which pipe to insert the heat. (SS) • If a data center is located without access to the district cooling grid, but with access to district heating, they can choose to invest in a heat pump and deliver first-rate heat into our system. The client will probably face investments in cooling machines, anyways. The best system solution is to invest in a heat pump instead, and sell the heat. (SS) • Today, we can’t account for deliveries of residual heat in our available capacity. This means that the deliveries go through whenever they occur, without us demanding full availability on that heat. If the supplier could guarantee the deliveries, we could account for them in our production planning, granting a certain value. Usually, heat pumps can’t provide high enough temperatures, requiring peak heat injection under dimensioning circumstances. (SS)

vii

• Usually, the limiting factor for capacity in the system is imbalanced flows in the supply and the return pipes. This implies that large production facilities, or data centers, must be located in areas with larger flows. (SS) • If a supplier commits to certain return temperatures in the cooling system, it will be reflected in the pricing. Our new price model for district cooling provides incentives for data centers to increase return temperatures. (SS) • When recovering heat, it is important to consider what temperatures and flows that can be permitted at each location. If there is a large pipe that passes with a large flow, the dilution can allow for lower temperatures. If there is a neighboring building that receives the recovered heat, temperatures cannot be allowed to be as low. (TE) • To receive residual heat on the return pipe for district heating will probably not be economically viable for us. Most likely, the customer will either deliver first-rate heat to the district heating supply pipe, or use the district cooling system. (TE) • Connecting data centers to district cooling is a good solution from all perspectives. We already have the heat pumps, which means we use existing equipment; it is the minimum possible investment and heat is recovered. However, the district cooling grid is not as extensive. Where possible, we will try to achieve a district cooling solution. (TE) • Connecting data centers for delivery of first-rate heat is similar to a cheap base load facility investment. To add one of the bigger data centers of 10 to 20 MW would be great. (TE) • Hourly prices for residual heat are desirable, but require more resources and are more difficult to manage. Seasonal prices are a calculated risk. For large facilities it would be possible to put more resources into hourly pricing, decreasing the total risk in the system. (TE) • For a data center, investing in a heat pump, the extra investment for delivering first- rate heat will be paid back on a reasonable payback-time. (TE) • We have a pricing model which varies during the year depending on the value of heat. (TE)

A.3.10 Synergies • We have already made large investments in heat pumps and infrastructure, which give us the opportunity to receive and cool residual heat from large facilities. We can easily take care of the residual heat in a rational manner through our existing system. The large gains come from optimally utilizing already made investments for the good of the total system. Of course, the utility must be big enough for both parties, for collaborations to be realized. (SS) • We have three different task forces working around the clock to ensure continuous operations of our systems. This means that we have as ambitious organization to handle our solutions. (SS) • When heat from a data center is recovered locally, the temperature requirement is often 10-15 degrees lower, depending on application. However, during large parts of the year the heating demand is low, consisting solely of hot water. One advantage by reaching the larger district heating system is that it contains a larger heating demand. (TE)

viii

• If a data center increases in size, it will soon have an excess of heat that the property cannot utilize. In that sense, it is an advantage to turn to the district heating system. (TE)

A.3.11 Obstacles • One obstacle for collaborating with data centers is the cultural differences and lack of understanding for the businesses of one another. I am sure that we are conservative in some ways, just as they are conservative in others. For example, we don’t see anything strange with liquid cooling, while it is terrifying for someone working in electronics. Another example is availability for cooling power. They might need redundant systems, and we are struggling with how to incorporate their heat deliveries in our production planning. All interfaces will be subjects for friction, and require work to bridge. (SS) • We can’t guarantee 100 % availability. On the other hand, I don’t think anyone can. The main cause for disruptions are power outages, which means that any data center without reserve power generation will be disrupted as well. We have the same availability on the district cooling as they have on their cooling machines, provided they have no reserve power generation. (SS) • Usually we come in contact with data centers when they have already invested in another solution, and in hindsight realized that they would like to recover the heat. We need to to enter the process at an earlier stage, to reach the best solutions. (TE) • The heat is valued higher if it is locally recovered compared to when it is sent to us. (TE)

A.3.12 Heat Recovery Limits • We have enough capacity to receive approximately 25 MW, during the peak summer, as it equals the total heat demand in the system. If we get 5 data centers with a cooling demand of 5 MW each, we have reached the upper limit of our current system. Certainly, the demand will change over time. (SS) • When the number of data center in the system increases, the risk is of all data centers shutting down decreases. We may not have to secure reserve capacity to cover for all of them. If we have a producer with a very large production, for example 50 MW, we might have abstained from strengthening production in that area, leading to increased risk. (TE) • There is pretty much always room to receive residual heat, up to a reasonable amount. Normally, if needed, we can even shut down the last heat pump, only using residual heat to meet the demand. (TE)

A.3.13 Potential Collaborations • We have entered a knowledge building phase regarding the potential for integration of data centers into our production. The next step will be to contact possible partners in the area. (SS)

Appendix B: Compensation Model for Recovered Heat The temperatures from data center operations are usually not high enough to feed directly into the district heating supply line. To match the supply line temperature, or at least decrease the gap, a local heat pump can be installed. With temperatures around 80°C, the heat can be fed into the supply pipe of most district heating systems. If the waste heat is cheaper than marginal production costs in the ordinary production units, the energy company might offer a compensation for the heat. The compensation level can be based either on the marginal cost of alternative heat production for the energy company, or the alternative cost of cooling for the data center. Regardless of what compensation model is used, the incentives from the energy company must never undermine the regular business. If the compensation is set too high, it might encourage the data center to invest in local recovery, effectively competing with the energy company. If the compensation level is too low, the incentives of investing in heat recovery are hollowed out. Thus, finding a feasible compensation level is not trivial.

B.1 Investment Analysis The investment horizon, cost of capital, and acceptable payback time for certain projects may differ between between energy companies and data centers. However, a theoretical experiment can aid in deciding a proper compensation level for the recovered heat. In this example, we consider an investment of a local heat pump at a data center in operation. The data center has an alternative cost of cooling, which will be affected by the investment. If they invest in the heat pump, they see their cooling costs change, and receive new economic contribution in form of a compensation. However, they will have to pay for the operation of the heat pump. The change in cooling costs comes from the different operation costs of cooling with the heat pump compared to the alternative solution. The operation costs of the heat pump is the additional cost for increasing the temperature to acceptable supply temperatures.

B.1.1 Acceptable Compensation Level For Data Center Heat Pump Investment A data center looking into investing in a heat pump, and starting recovery of heat may face the following equation for payback time:

� (��−∆��−��)∗ℎ/�� −�� + ∑0 � = 0 [01] (1+��)

�� = ℎ � 1 + ∆�� + �� [02] ∗∑0 � �� (1+��) The economic compensation for the energy company is given by:

� = �ℎ�� − �� [03] Where:

�� is the heat pump investment cost

�� is the compensation to tha data center for delivering the heat to the energy system

∆�� is the difference between operating costs of the old and the new cooling solution

�� is the operating cost of the heat pump ℎ/�� is the amount of operating hours for heat deliveries

�� is the cost of capital for the data center � is the desired payback time for the heat recovery project � is the economic contribution to the energy company

�ℎ�� is the market price on heat. The compensation level given by equation [02] for a specified payback time, t, indicates the break-even point for a data center to invest in a heat pump. If the data center is offered a compensation above, they should invest in a heat pump. If offered a compensation level below, they should not.

B.1.2 Acceptable Cooling Price For Energy Company Investment While the same investment in local heat recovery, but made by the energy company, looks like:

� ��ℎ�� +��−∆��∗ℎ/�� −�� + ∑0 � = 0 [04] (1+��) Where:

∆�� = �� − �ℎ�� [05] Resulting in:

�� = ℎ � 1 + �� − �ℎ�� − �ℎ�� [06] ∗∑� � �� (1+��) The cooling cost for the data center are now:

� = �� [07] Where:

�� is the heat pump investment cost

�ℎ�� is the market price on heat

�� is the revenue from selling the cooling service

∆�� is the cost difference between operating a local heat pump and the alternative heat production costs

�� is the operating cost of the heat pump

�ℎ�� is the alternative heat production cost ℎ/�� is the amount of operating hours for heat deliveries

�� is the cost of capital for the energy company � is the desired payback time for the heat recovery project

� is the cooling cost for the data center. Given a specified payback time, the energy company can’t offer a price on cool below the revenue from cooling, given by equation [06]. If the revenue from cooling is higher, they will profit from the investment. A span of feasible cooling prices is created, if the alternative cooling cost for the data center is higher than the lowest possible cooling price from the energy company.

B.1.3 Local Load Toughens Competition If a local load is connected to the data center, the data center has the option to sell upgraded heat at market price, effectively competing with the district heating company. If the data center takes on the investment of a local heat pump, the district heating company will probably not be a competitive alternative. A data center heat pump investment leads to lost customers for the energy company and uncertainty in excess heat deliveries, as the local load will benefit from being the prioritized customer. The excess heat deliveries will also occur more often in the summer, when the need is lower in the aggregated system, as the local demand is easier to meet at higher outdoor temperatures.

B.1.4 Mitigation If the payback time is met for a certain project, the data center might still be reluctant to take on an investment outside of their core business. Heat production, even if local, is the core business of district energy companies, making them more willing to undertake heat recovery investments. Energy companies have longer invetment horizon in general, increasing their possibilities of investing in projects with long payback times.

xii

Appendix C: Interviews with Data Center Industry Representatives

C.1 Interview Questions The interviews were based on following template, but adapted to the perspective of each interviewee.

C.1.1 General Who are you? What demands do you meet? What do you use your data centers for? Who are your target customers? What are the sizes of your facilities? (W) What are the limiting factors for growing the business? What are the primary risks for data center operations? Who has responsibility for energy-related issues in your company? - How is that responsibility defined? - How do you work with improving energy efficiency?

C.1.2 Economy What are the primary cost drivers in data center operations? - OpEx vs. CapEx? o The industry in general o Trends How do you consider investments? - Investment horizon - Risk

C.1.3 Market What market demands to you experience? What do the customers prioritize? What do you think about the market development?

C.1.4 Technology Reasonable assumption for outlet temperatures from data centers/equipment with or without heat pump? What do you think about the development of heat resistance for IT-equipment? What cooling technologies are used? What cooling technologies are preferred? How well do the temperatures in the district cooling system fit data center cooling demand? How does the load curve typically look like for a data center?

xiii

- Is it possible to adapt the load curve? Is it desirable? What are reasonable assumptions for heat density in data centers? In sections or total (kW/rack)

C.1.5 Heat recovery What do you think of heat recovery from data centers? How should a collaboration be designed? Who should have responsibility to remove heat when there already is enough heat in the system? Who should own the heat pump? Who has responsibility for cooling? What possibilities do you see in collaborations between district heating and cooling companies and data centers? What difficulties do you see in collaborations between district heating and cooling companies and data centers? What is needed from the district heating and cooling company to initiate a collaboration? How far are you (as an industry) prepared to go to start heat recovery collaborations? If already existing collaboration: What temperatures are delivered to the system? What tolerance levels do the different parties have?

C.2 Interview Responses This Appendix is a bullet point presentation of interview responses, thematically categorized and compiled. Each statement is followed by acronyms of the author, in parenthesis. The acronyms represent the following interviewees: JD – Jonas Dahlgren MF – Mattias Fridström MG – Martin Gierow LG – Lars Granström HH – Hans Havtun ML – Mathias Lindqvist JL – Jan Lundquist PN – Pelle Nilsson GS – Gert Svensson

xiv

C.2.1 Development on a Macro Scale C.2.1.1 Types of Data Centers • Large corporations and other parties with in-house data centers will outsource their data to cloud or colocation services. Both the cloud and the colocation markets are growing, as old data centers are shut down. (JL) • When a company needs to update old data centers, it evokes a discussion on whether the IT-activities should be outsourced. I think the new data center energy tax plays a part in this development, as other businesses can’t benefit from the tax reliefs. This probably pushes the transformation to more modern and environmentally friendly data centers. (JL) • Even the Internet giants need smaller data centers in the cities to get closer to their customers. Storing recently shown pictures locally and the rest in the mega complexes, is a form of edge computing. You want to come as close as possible to the customers, meaning that the smaller local data centers are a prerequisite for the large ones. (MF) • Streaming services like Netflix and YouTube are already distributed across Sweden, they are already running edge computing. (MF) • Except for some parties that rather keep their own equipment for reasons of confidentiality, IT operations is likely to be outsourced, if it is cheaper and better than the alternative. (MF) • In-house data centers will disappear completely. The time frame differs, so for some industries and agencies it will take a longer time. (PN) • The number of data center owners and the number of facilities will decrease as the industry is consolidated. There will be some large entities, delivering services to niche hosts. The data centers will be larger and fewer, to optimize capital expenses. (PN) • Even though each generation provides better capacity, the need for HPC computers will remain and they will be used for various simulations. (GS) • The cloud trend leads to a lot of large-scale data centers. These are supported by local edge data centers of some MW in size. The distribution is completed by micro data centers in power and cooling equipped cabinets at the client site. (JD) • There is a decrease in data centers in the span of 0-50 kW. They are outsourcing their activities to colocation. Most end up in facilities between 0,5 and 1 MW, and this is the segment that shows the greatest increase right now. Bigger data centers, around 10 MW, in the Stockholm region are usually made up by several colocated businesses. (ML) • The new data center tax applies only to commercial players. We are a governmental business, and KTH is not allowed to have subsidiaries. Counting all academic HPC computers, a lot of money could be saved in privatizing the operations. However, control will be lost and questions regarding ownership and operations will arise instead. (GS) C.2.1.2 Attractiveness for Investments in Sweden • Sweden has several advantages for attracting data center investments. As the energy mix is green, eco-friendliness does not cost you anything. Also, it decreases the risk of new taxes on non-carbon free energy to be imposed. The cold climate is good and the risk of natural disasters is close to zero. Other factors strengthening the Swedish brand

is the old tradition of high availability in power supply and energy-intensive industries, the political stability, and a world-class connectivity. (MF) • Being green implies no extra cost in Sweden. (ML) • The political stability in Sweden is an important factor for investments. With the new tax reductions for data centers, Sweden has the lowest electricity prices in Europe. The climate is good and provides a natural competitive edge and the risk of natural disasters is close to zero in Sweden. (JD) • New technologies for operating servers have improved the conditions for data centers cooling further south, in the middle of Europe. A cold climate is not necessary to the same extent as 5 years ago. Although Sweden is still considered to be on the European local market, the distance to Frankfurt is sometimes considered a problem. On the other hand, only 5 % of the data traffic needs proximity, 95 % can be handled if situated somewhere in Europe. Edge computing will solve the rest. (MF) • The risk of natural disasters is close to zero in Sweden. (JL) • The Nordic area or even Europe can be considered as the local region. Almost all deliveries in Europe can be made from Stockholm. (PN) • Sweden stands strong with low CO2-impact on the environment and low electricity prices. Also, the climate is beneficial for free cooling. (JL) • Facebook chose Luleå and Sweden because of electricity prices and the fact that Luleå had not had any power outages since the 80-ies. Facebook had planned for a large amount of redundant diesel fueled power generation capacity, but decided to trust the Swedish power system, minimizing. That was a great signal to the rest of the world. (HH) • In general Sweden has a very stable power grid, and I don’t think access to power will be a problem. (MG) C.2.1.3 Attractiveness for Investments in Stockholm • It is basically impossible to develop large scale data centers in cities. Especially in Stockholm it is hard to find the required power supply. Currently, there is not enough power and we can’t even be sure to have power when the ongoing power reinforcement program has been completed. (JL) • We shouldn’t build large-scale data centers in Stockholm, we should build them in the North, closer to the power production. (JL) • Most likely, we won’t see a competition for power in the future, but the risk increases with the closure of nuclear power plants across Europe. Nevertheless, I believe that the government is in phase with reinforcing the power grid. (GS) • The large-scale facilities require so much power that the power supply system is under severe pressure. A customer of ours, with some 16 MW in Stockholm, received a proposal for another 2 MW, but lost the deal due to a 4-year lead time from the power distributor. (ML) • Sweden is consuming less energy than before. Large industries, for example paper mills, have moved abroad. This is an opportunity that the data centers can benefit from. However, as the grid has been constructed for large industries in the north, large facilities that want to establish themselves in the south will find that the grid is not dimensioned for that. (MF)

xvi

• Energy companies looking to attract data centers to heat recovery and district cooling should pursue the smaller and distributed data centers in the city. However, you risk losing economies of scale. (JL) • There are 30 MW projects in Stockholm so, obviously, it is possible to develop large- scale data centers in Stockholm. You just can’t do it anywhere. One way is to build on former industrial sites, where power infrastructure is already in place. I believe that the potential for construction of large data centers in Stockholm will be exploited. (JD) C.2.1.4 Servitization C.2.1.4.1 Servitization of the Data Center • The colocation business can be a mix of different services, from leasing data center floor space and server capacity, to offering hosting services. (LG) • Hosting businesses often rent pay-as-you -go services and only pay for usage of energy and hardware. It is a clear OpEx focused trend in the industry. (LG) • The measurements of energy usage in pay-as-you-go models are made with varying levels of transparency. If a data center only does momentary ampere measurements, the transparency is decreased and it is hard to know what the customer pays for. (LG) • Depending on the end customer, a varying level of servitization is relevant. Some demand the host to own its equipment to increase trackability and minimize risks. (PN) • There is an inherent conflict between a high servitization level of the data center and the security operations. A server may be rented for process of sensitive data, and when that server is no longer needed it may be a risk if someone else started using that server. (PN) C.2.1.4.2 Servitization of Suppliers to Data Centers • Data center suppliers work with different models to handle the large-scale investments. If the data center wants avoid the investment, they may lease equipment. (JD, ML) • Data centers can lease cooling machines or UPS devices, effectively buying the service cooling or UPS. The suppliers servitize to meet the demand for pay-as-you-go services. (ML) • Storage can be bought as an OpEx service from the hardware suppliers, and they will collect their equipment after the contract time. (PN) • For large-scale data centers, the supplier can even provide an overall solution. (JD) • If the data center suppliers provide an overall solution in facilities they own themselves, they compete with colocation businesses, which might prove to be problematic in the long run. (ML) • It is increasingly common for us take on the service agreements and the operating agreements. The supply agreements for residual heat deliveries are written between the data center and the energy company, but the data center signs an agreement with us. This agreement basically frees them from responsibility, should the deliveries default. They do not want to put themselves in that position. (ML) • Right now, we are designing our the first heat pump, designed to supply temperatures for district heating purposes. (JD)

xvii

C.2.2 Investment C.2.2.1 CapEx and OpEx • Our biggest cost item is the CRAY super computer, with an investment cost at about 100 MSEK. The operational costs during 4 years are expected to amount to 60 MSEK, of which 20 MSEK are attributed to power expenditures. (GS) • Some of our storage is on SSD, but we also have rotating discs for the large storage volumes. The CRAY super computer storage volume has a 5 PB capacity, and in our investment analysis SSD turned out to be a far more expensive alternative. (GS) • A colder surrounding is generally favorable for computers, but the temperature is regulated by economical aspects. Many places in the US don’t have the possibility to use free cooling, and they will have higher operational expenditures on their AC units, regardless of fuel prices. (HH) • The difference in investment between free cooling and heat recovery are the heat pumps needed to enable heat deliveries into the district heating system. (HH) • The cooling efficiency could be improved immensely by drenching the computers in certain boiling liquids. Then you can make use of the vaporization heat. The only catch is the price of such liquids. (HH) • The investment costs for heat recovery are not a significant item, given that it is planned for from the beginning. An ex post investment is far more expensive. (GS) • A high security level on the facility increases costs. Data centers with less sensitive data usually scale down the security. (JD) • No one wants to own a server these days. It is an operational advantage if the data center owner provides the servers, as the data center then knows its devices by heart. I favor the ongoing trend of delivering services to customers, given the new business opportunities. However, it is a large investment to own several thousands of servers. Investments bind capital, but the staff and other operational costs are a bigger cost item. The currently low interest rates reduce the capital costs and the impact on profit, but it is harder to finance. Financing was easier when the clients owned their own equipment, because it could be bought through a leasing company and operated by us. Without the client investment, we need to handle the financing with securities on our business. (LG) • It is a clear OpEx-focused trend in the entire industry. The will to spend CapEx declines for each year for both power and hardware. Colocation actors can’t avoid the investments, but pay-as-you-go models are gaining popularity and the contract time frames are decreasing. The exception is handling of sensitive data, which usually entails CapEx. We need to know where all data go, with the implication that we need to own some equipment and sell OpEx services to our clients. There will always be a struggle to lease all the capacity of the CapEx equipment. (PN) • For Internet giants, it is all about Total Cost of Ownership. (MF) • EcoDC can make the cooling process cheaper than district cooling. The marginal cost of cooling a data center compared to an office is very high during the summer, as the cooling machines of an energy company only operates a few hours per year. (JL) C.2.2.2 Investment Analysis or TCO • We always analyze the total cost of ownership (TCO), looking at both the initial investment and the expected operational costs during the lifetime. The lifetime is

xviii

usually 10 to 15 years, and in that perspective, the capital costs are only a small part of the total. We are extremely focused on calculating energy efficiency and, even though the electricity prices are low today, energy efficiency drives significant cost savings. (JL) • It doesn’t cost you anything to be environmentally friendly, if you plan for it in advance. An ex post investment does, however. Buying 8 fans, when the facility is dimensioned for 4, may still end up saving you money over time, as the fans can be operated at lower RPMs and, therefore, decrease the power consumption. We buy expensive and over-dimensioned equipment, and still enjoy rather short payback times. To be environmentally friendly is not expensive, it is smart. (JL) • Initial investment cost is very important. Energy efficiency does not seem to be of highest priority, considering the design of the facilities. Many actors pursue low indoor temperatures and do not seem to dare to be energy efficient. (JD) • Some larger projects look at TCO over 10 years, but for most of the smaller projects, only investment costs seem to matter. However, TCO is becoming increasingly important, given the savings enabled through higher operating temperature levels of modern equipment. (JD) • There is a major focus on investment costs, although colocation actors are more aware of operational costs. Usually, our clients build their business model on a low PUE and then we need to achieve that level as quick as possible for them to start making profit. In that context, to bring forth a heat recovery project often leads to reactions on investment costs and the statement that district heating is not a part of their core business. But the advantages of having close to zero cooling costs are of course appealing. (ML) • The industry is divided in regards to investment analysis or TCO. Our clients claim to have extremely short payback requirements, but still have 10 year contracts with their landlords. This discrepancy opens for longer payback times, which favors heat recovery project with payback times of 6 to 10 years. The operational costs are appealing, but the investment is usually too large to bridge. (ML) C.2.2.3 Investment Horizon • The payback time on the CRAY computer is 4 to 5 years. With time, the capacity becomes insufficient and, by then, a new power efficient model will be available at a lower TCO. We use the same depreciation time for both computer and storage. (GS) • We have the opportunity to have a rather long investment horizon, having stable and long-term clients. We can make investments on more than 10 years. Our client base has enabled us to invest more in our facilities. (LG) • Most of our equipment is depreciated over 3 years, but even hosting companies depreciate infrastructure and network equipment over 10 years and more. (PN) C.2.2.4 Modularity and Scalability • We have bought 4 facilities, dimensioned for 6 MW each. We build one facility at a time and they are, in turn, modular to enable step-wise installation. When the first facility is fully occupied, we start building the next, and so on. (JL) • Our facilities are situated so close to the district heating connection nodes, that we install a fully dimensioned connection pipe from the beginning. It is more expensive to over-dimension for the early demand, but it is mere a minor post in the TCO. The

xix

equipment that upgrades the temperatures to fit the district heating system will also have a scalable design. (JL) • Data centers try to build modular but, usually, don’t start the building process until they have a certain amount of signed contracts. It may take years to fill up the facilities and the equipment is not optimized for part-load. You don’t want a data center running on 0.5 MW, if it is designed for several MW. (JD) • The equipment is always slightly over-dimensioned. You must install enough capacity to handle maximum load in all racks. In reality, maximum load is never reached. (JD) • You always strive for a higher occupancy level in your facilities. Also, we have a growth target as a long-term survival strategy. Economies of scale are forcing the smallest actors to exit the industry, as they can’t compete on prices. (LG) • Data centers need to be good at estimating when they will out-grow your facilities and expand in time. There is a risk of having a capacity deficiency. Instead, facilities are sometimes built, fully equipped with cooling and racks, and then left empty for years. (ML) • A common industry trait is that customers only use a part of the rented capacity in data centers. This leads to inefficiencies for the colocation businesses, as they have dimensioned and optimized infrastructure and equipment for a certain load. As the capacity is already sold, they can’t take in additional customers, but must expand in new facilities. If the equipment isn’t scalable, the colocation actor may find itself in a suboptimal situation. (ML) • Colocation actors never reach a rented occupancy above 85-90 % of total capacity. Simply put, it is too hard to find the right customers. Even all space would rented out, the load would never reach dimensioned capacity. (ML) • There are numerous facilities, dimensioned for several MW, currently running on idle mode. Eventually they need to be shut down. (ML) • Large-scale facilities have long investment horizons. Internet giants usually don’t change equipment; they buy more and expand into new facilities. (MF) • Usually, colocation actors build stepwise and scale up in size as the client base is growing. (JL) • The actors need to have agreements with power companies regarding power supply, which is a limiting factor for scaling a business. (HH) • Heat pumps could be installed instead of another chiller, when expanding. (ML) • Surface area does not seem to be a limiting factor for data centers, as storage is much more space efficient with SSD technology. (JL) • Client attraction is the limiting factor for growing the business. (JL) • Recruitment of competent staff is our main concern for growing the business. (LG) • Lack of capital is an increasing problem for the entire industry. The pay-as-you-go trend requires colocation-providers to own the servers, which means they need to make the investment instead of the client. Even with a payback time of 2 to 3 years, this leads to financing dilemmas. (LG) • For pure hosting businesses, the only limiting factor is the client attraction, as hosting companies only rent space and capacity from colocation-providers. (PN) • Internet giants usually have a desired design of data centers which the suppliers must adapt to. That implies custom solutions, but the lack in flexibility may lead to suboptimal energy solutions. (JD) xx

• The Internet giants have pre-made designs of data centers and leave the pricing to us. However, we would rather design and plan ourselves. (ML)

C.2.3 Availability and control C.2.3.1 Tier Classification • We don’t have a tier classification, but adapted the design to our specific operating conditions. We have focused on reliable equipment and infrastructure: electricity is reliable, district cooling is not; fixed components are more reliable than rotating components, etc. (GS) • We have UPS devices, diesel fueled power generators with N+1 cooling units. Our oldest CRACs have compressors with urban water. (GS) • Our ambition is a tier 4 certification, which requires continuous cooling. You need to be able to cool the facility, even with equipment failure or other problems. This calls for a complete redundancy of all equipment. (JL) • We don’t have a tier certification. I am slightly skeptical to commercial entities, like Uptime Institute. They have missed out on several important aspects for reliable operations; they have an extreme focus on technical redundancy, but lack in continuous risk assessment. The studies we base our security work on show that sabotage or human error are the main causes for incident reports. Especially since UPS devices have been subjects for major improvements. (LG) • Just like PUE, it is all about defining the system boundaries. Depending on the boundaries, your facility can seem more secure than it is. (LG) • Some facilities have fantastic redundancy, which is the focus for the tier classification system. When we constructed our high-security facility, we concluded that 80 % of reported downtime is caused by sabotage, human error or natural disasters. We decided to prioritize protection of equipment and education of staff instead. (LG) • We have noticed that the most security oriented clients don’t bother at all to look at tier classification. They evaluate our methods and usually send their own experts to inspect the facilities. (LG) • Just like with the PUE definition, industry people sometime refer to tier classification in a misleading way. (LG) • It is common that data centers don’t live up to the requirements of the aspired certification. Sometimes it is perfectly designed inside, but still have flaws on a system level. (ML) C.2.3.2 Redundancy • We need to have the redundancy in our system. We only see the power distribution from the national power system as a cheap alternative to producing our own electricity, but we can’t consider it the primary mode of operations. If the power line is cut, we must manage on our own, even with failures on some of our reserve units. (JL) • For district cooling to be appealing, you need to be able to see it as a cheap alternative, and not the primary production system. (JL) • Colocation clients often need to be educated on the operations and risks. (JL) • The investment costs differ a lot between different availability levels. If you pursue a high classification, your investment costs will go up. (JD)

xxi

• You need some sort of redundancy in the facilities. Otherwise, you will be out of business if you suffer a cooling unit failure. Also, you want to be able to service the equipment without shutting everything off. If you are only pursuing a low availability classification, shutting off the equipment might be unavoidable. (JD) • I don’t think that one of the Internet giants would connect to district heating or cooling without a cooling option of their own; it is only possible as a complement. (JD) • We have full redundancy on our facilities. Our power supply is doubled: two diesel- fueled power generators, power distribution from two different directions, double transformers, and double switch-gears. In one of our facilities, the system is even more redundant, and at least one of the sets has a perimeter protection to prevent sabotage. (LG) • 99 % of the time, we have the super-efficient cooling solution from our property owner, and even though our alternative option is not environmentally friendly, it is irrelevant in the sense that it is rarely used. (LG) • The Internet giants don’t bother to certify their facilities; they build redundancy by setting up additional data centers and mirroring the data between them. That option is, of course, only available if you have lots of money. (MF) • Redundancy is expensive. There are still several high security projects under development, that haven’t started operations yet. Perhaps the clients aren’t prepared to pay for the top certification levels. There are several middle segment actors for customers to choose between. (MF) • MAX IV have a complex system with duplicated pumps and a backup cooling machine if the heat pump would fail. The heat pump is divided into several smaller units. (MG) • Clients usually have a requirement specification accompanying their orders, but the quality of the specifications vary with the organizational maturity. It would be easier if the clients would use international standards to a larger extent. That ranges from security classifications, to environment classifications, to tier classifications. (PN) C.2.3.3 Risks and Risk Assessment • We try to make recommendations to clients, but they often have a list of requirements for the design. This goes for both tier classifications and ASHRAE-levels. Certainly, the legitimacy for some requirements could be questioned. There are some environmental requirements, but the main investment driver is security. (ML) • We minimize our risk by working with partners. (JD) • A common problem is that data centers have a lack of competence on their staff, to run the facilities around the clock. (LG) • Both in-house and outsourced operations are subjected to risks connected to staff. The difference is that you can make precautions and take action if you have it in-house. The smaller the organization, the easier to control the information flows. (LG) • The risk of someone being corrupt in your own organization is equally big as in a data center. (MF) • The dimensioning risk for our business is if we would start leaking client data. That would be the worst-case scenario. (PN) • Shutting down machines is always a risk, although it has decreased dramatically with the introduction of SSD. Nowadays, practically everything is stored on SSD. (PN)

xxii

• The main risk factor is customer attraction. You need to take an informed risk, when building a large-scale data center without all clients signed up. (JL) • Cooling units, power generators and other equipment need to be on the surface, and if that was to be cut off, the whole facility would stop working. It doesn’t matter if the server is situated in a bomb safe room. (JL) • There are deep underground facilities with two separated power and cooling distribution systems. They have a high security facility, but the surroundings are rarely included in the system risk assessment. (ML) C.2.3.4 Owner Structure and Control • I would never trust the property owner and their pumps, even though they have redundancy. Their organization is not dimensioned to operate data centers. We have a good property owner, but I find the cultural differences hard to bridge. (LG) • If a data center engages in a heat recovery solution, the system must be predictable. Day-to-day variations can’t be tolerated. As long as that criterion is met, our clients don’t find collaborations problematic. (ML) • If the data center doesn’t want to be a district heating supplier, they can always find someone to own and operate the heat pump. But then it can be problematic to account for the pump as redundancy. (ML) • There are recent examples of when data centers have chosen district cooling as one option, and heat recovery as another. (ML) • I don’t think data centers would want anyone to operate their cooling system. They only write service contracts with trusted actors. (ML) • You want local capacity, so the first-hand choice is always to own your own heat pump. You got to have your own capacity, to be able to guarantee and control the cooling. District cooling can’t be counted as redundancy, as it is out of your control. If you could lease capacity on someone else’s pump and it could count as redundancy, as you still control the cooling, yourself. The placement of the pump is not the main concern. (PN) • If the heat pump is kept at the data center, the energy company will have nothing to do with it, even if they own it. For example, they might have a service agreement with us, the supplier. (ML) • Some data centers prioritize control of the temperature before heat deliveries, and therefore own their own heat pumps. (HH)

C.2.4 Energy and Cooling C.2.4.1 Power and Heat Density • High Performance Computing (HPC) clusters performing complex calculations have high power density in their servers (GS, JL, LG) o Research on turbulence, fusion, climate and the human brain are examples of research with such complex calculations. The demand for computing capacity is high, and much more than is currently available could be used. (GS) o A normal calculation program operates at approximately 80 % of max capacity and if the computer is idling it might use 25-30 %. (GS)

xxiii

o At the Parallell Data Center at the Royal Institute of Technology, there is a queue to get access. Commercial companies pay to get access to computing capacity. (GS) o There are HPC-actors that claim to reach 200 kW/rack. (JL) • Bitcoin miners are always competing to win the next bitcoin. This means they need the best equipment with the fastest processors. As bitcoin miners need to continuously update their equipment, their investment cycles are shorter. (MF) • The power density is on average not as high as people expect. My estimation is 6 kW/rack. It is very difficult to predict a future value. A couple of years ago, I would have guessed on higher numbers. but the modern servers are more energy efficient than before and they are operated more carefully. The heat density will probably increase, but not as dramatically as was believed a couple of years ago. In 5 years, the power density might be 8 kW/rack. (JL) • Data storage is increasing dramatically, but will probably not increase power density much, as there is a technological development in SSD. (JL) • Our average power density is 2 to 3 kW/rack, or between 5 and 6 kW/rack, if the equipment is composed by us. Customers renting server capacity often believe their applications consume more power than they do. During procurement, the client may request a power and cooling capacity of 6 to 8 kW/rack, while the real demand for the applications only amount to 2 to 3 kW/rack. A common misunderstanding in the industry leads to customers asking for a capacity corresponding to the demand of their old equipment. Also, they may add safety margins to enable more applications. We measured all our servers that have power supply units of 1.2 kW and concluded that they on average consume 158 W. (LG) • When dimensioning power supply units, it is important to consider that the starting current is much higher than the operating current. There are also other factors which increase the power requirements on the power supply units. (LG) • In general, data centers do not utilize even close to the design server power density. (ML) • The average power density of server racks is 5 to 10 kW/rack, depending on the age of the data center and how much power that can be distributed to the rack in a proper way. It is also dependent on the cooling system. However, the trend is that power density of servers is increasing. The heat density will grow with the efficiency of the data centers. (PN) • A higher power density in servers could be achieved by increasing the number of clients utilizing the server, implying increased revenues per server. (PN) C.2.4.2 Liquid Cooling • Water cooling is not a new technology, but there has been a lack of economic incentives. (GS, HH) • Liquid cooling is more efficient and achieves high return temperatures, around 50 oC. (LG, HH) • There are solutions for liquid cooled servers. It is possible to achieve liquid temperatures up to 50 oC, rendering compressor cooling unnecessary. All that is required is a liquid cooler outside to dispose of the heat or a circulation pump to deliver it to the district heating system. (JD)

xxiv

• If the return temperature of water is 40oC, free cooling would be applicable in the warmest places in Europe. (GS) • Liquid cooling is quiet. (GS) • Liquid cooling is space efficient. (HH) • In the case of liquid cooling, the data center industry is afraid and conservative. Liquid cooling is at least as secure as current standards and has been around since the 90-ies. The difficult part is to get the liquid onto the electronic components, enabling the largest energy savings. (HH) • Liquid cooling is foremost relevant for a specific market segment – High Performance Computing (HPC), used for power intensive applications. It is not relevant for colocation. (JL, JD) • With an increase in HPC-clusters, big data and data mining, there will be an increase in data centers directly cooled by water (JL). • For server rack densities of 10 to 15 kW/rack, it is cheaper with air cooling. It will be expensive to build liquid cooling where it is not actually needed. It requires specially made servers and server racks and 2 to 3 heat exchangers in each rack to exchange the heat from the refrigerant. (JD) • No air cooling technologies of today can manage 40 kW/rack. (ML) • The server manufacturers will instigate the development of liquid cooling. Suppliers of infrastructure to data centers will need to adapt to the servers provided to their customers by, for example, Dell and HP. However, the technology does not feel safe enough just yet. (ML) • It is most likely not the actors from heat recycling initiatives that will influence the market to become liquid cooled. (ML) • Liquid cooling is something that needs to be overcome emotionally. Data centers do anything to protect against leakage. (LG) • Liquid cooling is a cost issue; it will be more expensive. (HH, JL) • Liquid cooling is about economic incentives in the case of HPC. In the case of hosting and colocation, however, customers bring their own IT technology. The data center will then have to apply standard components, and it is currently not standard for components to be liquid cooled. (GS) • Air cooling would not be applicable at such high-power densities as for the HPC computer at PDC. (GS) • The CRAY computer cooling works like the rear door-cooling technology. Then there are other liquids than water, such as Freon liquids. There are both liquids that boil and liquids that remain in only one phase. It is possible to cool the door with water and have another refrigerant inside, if one is afraid to use water. Liquid can be boiled in a coil or the entire computer can be drenched in the liquid, which will then boil. (GS) • Liquid cooled servers can be more forgiving and suitable for heat recovery even when power density in a data center is low and varies, as it is still possible to achieve high return temperatures. When air cooling is applied in such a data center, different temperatures mix and it is impossible to achieve such high return temperatures. (LG) • When meeting with suppliers of liquid cooled servers, the technology still feels a bit immature. However, they have rather flexible systems to drain each machine, when needed. (LG)

xxv

• Our sales force is very skeptical to water cooling, as we don’t dare to guarantee the safety of it. (ML) C.2.4.2.1 Liquid Cooling of Components • Cooling on components could prolong the service life for the component. (LG) • The temperature of the components affects the silicon, which ages faster if the temperature is elevated. (HH) • Liquid cooling of components could allow for a higher operating frequency for the processors, without overheating. (GS) • Liquid cooling of the components is more expensive, which require that the user can take advantage of the increased temperatures of the refrigerant. Component cooling will be relevant for future generations, but subjected to economic considerations. The economic incentives to raise the temperatures to 50 degrees were not present prior to investing in our current equipment. (GS) • Liquid cooled components require an entirely different operating organization. When a company invests in such a technology, it must have an operating organization that can manage it, as service will entail an increased risk. It is difficult to motivate the cost of such an operating organization if, for doing predominantly routine maintenance. (LG) C.2.4.3 Temperatures • Required supplied air temperature for cooling modern data centers is: 20 to 22 oC (HH), 22 to 23 oC (JL), 25 oC. (JD) • If the temperature in the data center is between 10 and 15 oC, the applied cooling technology is outdated. (HH) • Returned air temperature vary but could reach 40 oC if supply air is 25 oC. The difference in temperature between air inlet and outlet is 10 to 15 oC. (JL) • Our cooling system delivers 20 oC to the heat pumps. (LG) • If the temperature of the return air should reach 37 oC, you must accept 27 oC for the air supplied. (JD) • The outlet air could be between 35 oC and 50 oC (HH) • Some actors probably operate their data centers too cold rather than too hot. In the last few years, the accepted operating temperatures have increased to 22 to 23 oC. When the next ASHRAE recommendations arrive, the temperatures will increase further. Operating temperatures of 40 oC are an issue of work environment rather than technological limitations. (JL) • Temperatures in the cooling system are usually 15 oC in the inlet and 23 oC in the outlet. Some data centers have 20/30. It is desirable to increase the temperature of the refrigerant, to achieve as much free cooling as possible. However, actors might not dare to increase it enough. (JD) • The temperatures offered by district cooling, 6 oC inlet and 11 oC outlet, are too cold. It is enough with 16/22 oC or 16/24 oC. We have much better cooling equipment in a data center compared than in ordinary office buildings. It would be more beneficial to receive the return temperature of the district cooling system. (JL) • The temperatures provided by district cooling is a perfect fit for us. (LG) • Regarding suitability of the temperatures provided by the district cooling system. The three cooling technologies discussed below are all used by KTH PDC (GS):

xxvi

o For Computer Room Air Conditioner (CRAC) units the supplied temperature is fine, but the return temperature is a bit high. Traditionally, the temperature in the data center is around 20-22 degrees which will require rather large heat exchangers to reach 17 degrees for the return water. o For encapsulated cooling it is easier to achieve 17 degrees as return temperature since the temperature inside the hot aisle is approximately 30 degrees. o When water cooling is used, the return temperature of the water exiting the computer might be 30 degrees, in which case it is a waste to mix it with 17 degrees. o Future system may achieve higher return temperatures of approximately 45 degrees. o Another aspect is condensation. 5-7 degrees water will condense at certain relative humidity levels. Especially with water cooling this might want to be avoided in the computer. It is possible to install a heat exchanger elevating the temperature but a high difference between supply and return temperature limits the amount of water. o To conclude, the current temperatures for district cooling are needed for older installations but a bit higher temperatures would be more effective (especially for the energy company). (GS) • Regarding suitability of the temperatures provided by the district cooling system (ML): o The supply temperature is lower than needed and in many cases the temperature in the return pipes would have been enough (which would also contribute to the district heating production). o However, for the systems for comfort cooling, which are also supplied by the district cooling network, a temperature of around 7 degrees is necessary o I imagine data centers could be connected to the return pipe for district cooling in the future. (ML) C.2.4.4 PUE • PUE is a bad measurement. (HH) • PUE is important, equally important is transparency in the calculations. (JL) • PUE only works if one takes responsibility for the entire chain. It not actually adapted to data centers buying district cooling or applying heat recovery. Strictly speaking, heat recovery should give a worse PUE-number if heating pumps are used to increase temperature of the refrigerant. It is very important what one chooses to define as the total system and where the system boundaries are. (LG) C.2.4.5 Load curve • The load curve for most colocation actors is flat on a day-to day basis (PN, LG, ML) • The HPC computer has a relatively flat load curve throughout the year, with small fluctuations when loading new data sets. (GS) C.2.4.6 Established Cooling Technologies • Hot and cold aisles is a common cooling technology. (PN, LG)

xxvii

• By blowing air from below the floor, it is easier to adapt the cooling depending on the customers’ equipment. (LG) • Most actors have some form of air cooling. Some have liquid cooling built into the computers, but there are still fans that transport the heat from the components to the liquid. No commercial data center utilizes direct cooling of components. (HH) • Elevated temperatures in data centers increase potential for energy efficiency. It enables apply air-to-air heat exchange without compressors. (JD) • Usually, air in a data center doesn’t circulate as assumed but is affected by turbulence. (HH) • The applied solutions for air distribution are very inefficient. (ML) • The fans in in-row cooling operate at higher RPM than larger fans operating in the room, using more energy. As the units for in-row cooling or cooling in the aisle are under-dimensioned, the fans must operate at high frequency which causes high pressure drops in the cooling circuit. (JL) • In-row cooling is more expensive in regards to the initial investment. (JD) • In-row cooling achieves higher precision and is easier to service. If one part breaks, everything does not have to be serviced, and the data center becomes less vulnerable. (PN) • It is rather inexpensive to build redundancy with fans. (JD) • It seems that more actors are moving away from direct acting cooling, i.e. bringing outside air directly into the facility. It causes problems with humidity and pollution (JD) • The data center operator measures the temperature of the return air and regulates the temperature of the air that is supplied. It is adjusted continuously depending on how intensely the components are operating. It is important to keep an even temperature in the equipment. (PN) • 15 years ago, consensus was that it was supposed to be very cold in data centers. Now, it is 25 oC all year around. (PN) C.2.4.7 Energy Efficiency • It is more important for the CEO of a company to be environmentally friendly than for the IT-department. (JL) • The “green” IT-perspective will increase (JL) • Site managers are responsible for the energy bill sent to the customer. They also directives for energy efficiency from management. One directive could be to decrease the PUE-value. A low PUE-value ensures a high level of effective loads. We continuously set new goals as a part of our environmental and I, as CTO, am ultimately responsible for energy efficiency in our data centers. (LG) • I, as head designer, am responsible for that the data center is built energy efficient. (JL)

C.2.5 Heat Recovery C.2.5.1 General Attitude Towards Heat Recovery • Heat recovery is a win-win and could affect the choice of location for a data center. Previously it has been popular to put data centers in regions with high availability of

xxviii

free cooling. Many colocation actors think Stockholm is a good location and for them selling heat is appealing. (GS) • Heat recovery is a win-win for all involved parties. (LG) • If it is possible to make use of the energy from data centers, it should be recovered. New industries are welcome into the district cooling system. We have built a heat powered cooling production in the cooling grid too counteract a mismatch between demand and supply. (MG) • There are opportunities for heat recovery, especially if it is possible to control the injection points in the district heating system. (HH) • Collaborations on heat recovery is an issue of price. We still pay to get rid of the heat. For us to switch systems, total cost must decrease. (LG) • It is important that the heat is recovered to be used for something. We are prepared to pay to keep our image and values. (LG) • Heat recovery is easy to understand for the energy companies, but very difficult for the American data centers. To convince them, you need to be overly pedagogic with nice graphics and simple explanations about how it would work. The concept of being paid for residual heat is unknown to American companies, and in that aspect communications need to be improved. (MF) • Having a green business is always interesting, but the main driver is cost of ownership; money. (MF) • Heat recovery increases profit per leased rack space. (ML) • Several of our clients are interested in heat recovery possibilities, but it can also be a challenging to convince the energy company. (ML) • The Internet giants, like Google, would never be able to recover all residual heat, since the heating demand on their sites is too small. (ML) • Usually, the requirements are general in nature and we have yet to see a specific demand for heat recovery. An environmentally friendly profile and green energy input are taken for granted, and the customers don’t expect a premium fee for that. There are probably customers of data centers that are prepared to pay extra for environmental solutions. The requirements come from the end customer and moves along the value chain and forces the suppliers to adjust accordingly. (PN) • The large data center actors are currently not attracted by heat recovery. (JD) • The incentives for heat recovery decrease with cheaper electricity. Free cooling is cheap as well as operating cooling machines and compressors. There is normally no residual heat from cooling machines. (JD) • I think data centers are prepared to pay for achieving an environmentally friendly profile. (JD) • There is no money to be made in delivering water at 25 oC. (JD) • There is a mismatch in regards to the property owner not operating with the same uptimes and reliability as a data center. If a property owner had had the same requirements as a data center, an emergency cooling system would not have been needed, and that investment could have been placed elsewhere. (LG)

xxix

C.2.5.2 District Cooling • We have a very high availability of district cooling, around 99.9 %. We have three large heat pumps, one district cooling accumulator, two cooling units, and duplicated distribution pumps. A lot of things must fail for us not to produce cool. (MG) C.2.5.3 Established Collaborations – technical configurations • The HPC computer at the Royal Institute in Technology in Stockholm recycles heat to the university campus. This is managed by the landlord, Akademiska Hus, which also provides the connection to Stockholm’s district heating system. Akademiska Hus sometimes purchase heat when it is difficult to produce on site. KTH PDC provides Akademiska Hus with heat for approximately half of its consumption. (GS) • The heat flow into the district heating system is not separated by temperature. Hotter water is produced from the CRAY computer and in the future, it will probably be even warmer. It is a shame to mix it with water of 20 oC. I have lobbied to install separate piping or place the heat pump here, but today the heat pump is in a building a short distance away. (GS) • The heat pumps elevate the temperature to 60 oC so it can be used for heating purposes such as tap water. (GS) • The first attempt of heat recovery from the HPC computer produced water at 30 to 35 oC, which could then heat the chemistry building without heat pumps. The computer still produces water at 30 oC is but now it is mixed with the cooler flows before entering the heat pump. (GS) • We have three different cooling systems. The primary cooling system is connected to a borehole heating storage. The redundant capacity is a connection to district cooling. Additionally, if both systems would fail, there is an entirely separate piping system with urban water and compressors in some of our cooling machines in the data center. When the property owner invested in the borehole heating storage, it was concluded that we would operate data centers for a long time, and that we would be a reliable heat supplier. The emergency cooling system with urban water has only had to be used a couple of times. (LG) • Before the borehole storage, the heat was also recovered locally when cold outside. However, there was a lot of excess heat during the rest of the year. (LG) • We deliver district cooling to data centers and the heat that is cooled off by the district cooling is brought back to the district heating system. (MG) • MAX IV is a facility which requires specially adapted production of cooling. We deliver cooling to different equipment on three different temperature levels. It achieves higher efficiency compared to if cooling was delivered directly from the district cooling system. In addition to this, the facility is located a bit far from the district cooling grid, so we have built a local cooling grid which only produces cooling to MAX IV. However, the facilities recover heat to the district heating system. (MG) • We control the heat pump at the MAX IV compound and are responsible for providing continuous cooling to the facilities. From the heat pumps at MAX IV, we supply district heating at 80 oC, which is sufficient for the buildings in the area. (MG) • Some data centers consider green initiatives to be very important. One data center has installed 200 kW of solar panels with an investment cycle of over 30 years. Furthermore, they are determined to recover their residual heat. (ML)

xxx

• When data centers are located far away from the production site of the energy company, it can provide a beneficial contribution to the system. (ML) • Some energy companies receive all their heat from residual heat sources, and thus have very low prices for district heating. They have basically no purchasing cost except for energy to the heat pumps. In regards to heat recovery, they cannot pay more than what they sell for. If the sources disappear, they will be left without residual heat and forced to review their district heating prices. (ML) C.2.5.4 Heat Recovery Technical Configuration Possibilities • It is important to separate the cold and the hot side of the cooling system in a data center. (HH) • Organic Rankin Cycles have very low efficiency. Even if the heat is free, the system is not. (HH) • It is not possible to only have a direct connection to the district heating system. District heating and cooling can only be a complement to the existing cooling system. A redundant circuit which leads to chillers or a refrigerant cooler is required if the system stops working. If a heat pump is installed, it can be viewed as the primary system. (JD) • We have discussed a couple of business cases where the requirement is 80 oC for supply to district heating, but it varies depending on where the data centers are located in the grid. (JD) • It is possible to have planned breaks in residual heat deliveries, which have been agreed upon beforehand. Also, the allowed temperature for delivered heat may depend on outside temperature. (ML) • In one case, we have achieved district cooling on one leg and district heating on another, resulting in the local energy company as the sole supplier of cooling. (ML) • Allowed temperature of delivered heat could be adapted to the mixing temperature in the specific pipe. (ML) • The district heating companies work hard to lower their return temperature in the district heating grid, and are therefore opposed to increasing it. (ML) • Many district heating companies have too high temperatures in their supply lines. It could be because of short circuits in the grid, because a lot of customers have broken systems, or because they built the system in that way. (ML) • If the temperatures from the data centers are too low, then it is not profitable to make first-rate heat deliveries, as the efficiency will be too low. In that case, it is better to deliver lower temperatures. (ML) C.2.5.5 Established Collaborations and Responsibilities • KTH PDC manage their operations on their own premises, and then there is a heat exchanger at KTH which is controlled by Akademiska Hus. Akademiska Hus manage the heat pump and deliver cooling. The heat pump also delivers comfort cooling to other parts of campus. (GS) • Today, KTH PDC pay for cooling. There is an ongoing discussion regarding the pricing of the heat deliveries. (GS). • A data center can either buy the service cooling or own a heat pump. If a data center owns the heat pump and sells heat to the energy company, it has to modify the system boundaries for the business. (HH) xxxi

• The heat production in EcoDC, used for the pellets factory, also produces cool. (JL) • We are a tenant but own the infrastructure in the data centers. It is impossible for us to make a 30 MSEK investment for a borehole storage locally, because that would make it impossible to move. In that aspect, it must be the property owner that owns the cooling solution. If you own a property, it is easier to make these types of long-term investments. To make them as a tenant makes no sense. (LG) • If you don’t own your own heat pump, you can’t control it. I would never dare to depend on the property, as they think they have enough redundancy by buying district cooling. (LG) • We perform continuous risk analyzes. That type of strategy does not exist within the real estate industry; it is not part of their core business. (LG) • When we considered a district cooling solution a couple of years ago, the variable cost per kWh was too high for that solution to be interesting for us. (LG) • The pricing model for MAX IV is formulated in such way that MAX IV is never paid more than the value of the heat in the system. (MG) • In one case, a property hosting a data center invested in a local heat recovery solution. They switched from buying both district heating and district cooling, to reutilizing the heat from the data center internally. Instead, they deliver heat into the district heating system during low-demand periods. (ML) • The suppliers to data centers are the ones who will approach them with heat recovery projects. (ML) • The party with the best interest rate at the bank will make the investment. There is one example where an energy company owns the installation, we installed it and the data center is operating it. (ML) C.2.5.6 Pricing Model • A large colocation business constructed a facility with possibility to deliver heat to the district energy system. The system is constructed to automatically calculate whether to operate o the district heating or cooling system, based on electricity spot prices. (JD) • If a data center is paid to deliver heat, it is only the difference between the price on heat and the cost for cooling which will be important in long-term. If what is paid during the winter would cancel out what the data center must pay during the summer, it would be interesting. (LG) • It easier for us to be able to operate by established agreements for heat recovery, instead of participating in a long processes to formulate new agreements every time. (ML) • In most of our collaborations, we have split the investment and the revenues with the partner. We strive for complete transparency of project costs and alternative costs for both parties. From those calculations, a fair split is derived. The transparency in the planning phase leads to deeper trust in the operating phase as well. (MG) C.2.5.7 Division of Responsibility • Data centers prefer the heat pump to be local. (GS) • Large heat pumps, 600 kW and above, might be operated by energy company. (JD) • A district heating company could be a better partner for cooperation than a real estate company as a district heating company has a 24/7 service organization and that kind of

xxxii

mindset. Energy companies have a number of technicians and engineers, so I think that is a better match. (LG) • Data centers are not keen to commit to delivering heat, it is foremost only nice to have. (ML) • To commence a collaboration, the district heating companies need to be a stable partner, which they already are. They have foresight, longer business cycles than data centers and are municipally owned. (ML) • We are not interested in committing a certain delivery of heat. Our business is not built that way; we guarantee a certain delivery and capacity to our customers. To commit to a certain delivery of heat, the data center business has to be built accordingly for the start. (PN) C.2.5.7.1 Servitization of Cooling • At the moment, we don’t operate heat pumps, but work a lot with partners. (JD) • We can have service agreements with the owner of the heat pump, regardless if it is the data center or the energy company. (ML) • If the energy company would like to own the installation and do the maintenance, they would have to undertake the whole cooling production, the filter services, and several different processes in the data center. Energy companies have specialized competence on large-scale heating and cooling, but IT is a niche environment. (ML) • In Stockholm, energy companies want to attract middle sized data centers for heat recovery. When there already exists a heat demand, the energy companies may find it profitable to servitize production. (ML) • Connection to district cooling is a much smaller investment and can be more relevant than a heat pump investment for an already established data center actor. The data center may not be willing to invest in an energy or environmental project, but might be more willing if it reduces operating cost. (ML) C.2.5.7.2 Servitization of District Heating Systems • If a data center in could sign delivery contracts with different energy companies, regardless of the system owner, that would turn the whole situation around for heat recovery into district heating systems. Currently, energy companies already have delivery contracts between each other. (ML) • The district heating market differs a lot from the electricity market, but it would be desirable to have the option of choosing between different district cooling suppliers. One wonders when it will be possible to buy the heating and cooling from one company, but the distribution from another. (LG) • When we started considering heat recovery from data centers, it was obvious that the data center would be responsible for producing the high value heat. However, that might be changing. At some locations, where an area is constructed to host data centers, the heat pumps may be owned by energy company. In other places, the data centers might keep ownership of the heat pumps. In that case, we deliver the required equipment. (JD) • The energy companies in the Stockholm region which do not partake in the servitization trend will fall behind. (ML)

xxxiii

C.2.5.8 Enabling Factors • One energy company has proposed to move KTH PDC outside of campus, to where there is a mismatch in heat demand and district heating production. There are locations where there is a larger need for local addition of heat production. Although it might be profitable for us long-term to put a new super computer there, KTH wants to keep the computer locally. (GS) • Residual heat from data centers is competitive in comparison with other fuels, even with an electricity powered heat pump. The possible payment for heat recovery from a data center is related to the marginal cost of fuel for heat production at that specific location. (JL) • For us as an energy company, it’s considered standard sales procedure to contact facilities with a high turnover of energy. (MG) • In regards to data centers, we have the advantage of an experienced operating organization, managing heat pumps all year round, 24/7. (MG) • Heat recovery is mainly presented as an energy project. Two years ago, economy was the only driver but that is starting to change. (ML) • I want data centers to not think of it as delivering district heating, but as receiving cooling. The dream scenario is to get as far in the cooling process that a heat pump is not necessary. Unfortunately, the district heating grids in Sweden are not built for that. (ML) • 55 to 60 oC output temperatures from data centers would be possible, should they start using liquid cooled servers. (ML) • There must be some kind of guarantee for a data center to be allowed to deliver heat a certain number of hours, for them to make an investment decision. (ML) • A lot of larger companies have an increased interest in being perceived as nice companies. If heat recovery contributes to the total system, it could be an appealing aspect. They are also interested in the possibility for receiving payment for the heat. (MF) • There is an opportunity to educate American companies regarding the concept of heat recovery. This probably requires a typical case which can demonstrate how it works. Stockholm Data Parks is a first step in creating such a typical case. (MF) • A lot was learned from the processes with Facebook in Luleå and also when Sweden lost Apple to Denmark. Viborg in Denmark was very good at making it easy for the buyer. These investors do not have time for difficulties or complexities. (MF) • An advantage of heat recovery is that the data centers gains revenues as the ability to use that feature for branding. (PN) C.2.5.9 Package Deals • It is possible to exchange one of the redundant cooling systems in a data center to district cooling. It is important that the energy company can offer a package deal which is operationally reliable from the start. (GS) • If the district cooling company presents a cost-efficient package deal with proof of concept, they can offer have a few standardized solutions instead. (GS) • To attract new establishments of data centers, there should be a complete set up with prepared sites. (ML)

xxxiv

• If it is large data center of several MWs, the site must be finished and partners should be secured to supply infrastructure to the site. Part of the investment needs to be made beforehand. (ML) • For a smaller data center, for example a colocation actor of 1 MW, it might be sufficient to have a preformulated deal to offer. It could also be useful to have a reference project. (ML) • It might be sufficient to only connect existing data centers. If the map of Norrenergi’s grid is placed on our map of our customer, at least 10 MW worth of data centers can be located. Then it is important to find suitable business models for them. (ML) • The district heating companies should approach the established collaborations. Those collaborations have the competence and knowledge of how to work with internet giants, and several of them have a regional interest or a national interest. The district heating companies could also contribute with knowledge. (MF) C.2.5.10 Obstacles • Heat recovery has been a buzzer for quite some time but, until now, there have been no success-cases to promote. (MF) • Both pricing and interface need to be regulated in a successful collaboration. (GS) • The difficult part is probably procurement and negotiation. One data center had problems getting finding a recipient for the heat, as the heat demand is lower in the summer. (HH) • Availability and the operational reliability need to be sufficient. Many energy companies have not understood what operational reliability means for a data center; one disruption a year may be unacceptable for some businesses. The district cooling in Stockholm is known for its reliability, but there are still several disruptions a month. If connected to district cooling, the data center needs backup or redundancy. One energy company solves the issue by enabling connection to both district heating and district cooling. (GS) • What is most important is that the energy company has understood that the data center operation needs service availability 24/7. The configuration needs to be dimensioned in accordance with pre-determined tolerance levels. For example, one energy company said that they send out 8 oC and bring back 18 oC. A data center was dimensioned accordingly, but when district cooling temperature arose to 8.5 oC emergency cooling was needed. The energy company claimed 8 oC to be a guideline. Better communication is needed between the parties to achieve understanding of one another. (GS) • In district energy systems, there is always an excess of heat during summer. (JL) • Payments for heat deliveries increase with temperature. However, regular heat pumps can’t deliver 100 oC, and can’t keep up during high demand periods. To reach higher temperatures, new machines and refrigerants are necessary. (JD) • Residual heat is of interest up to a point where it is no longer needed, so there is always a risk. Many suppliers of residual heat have an excess during the time of year when there is no demand. (MG) • The reluctance to becoming a district heating supplier is one of the largest obstacles to overcome, and sometimes we have to find a different solution where someone else

xxxv

takes on the role as heat supplier. Security, investment and supply requirements are three other obstacles that need to be overcome. (ML) • Heat recovery is nice to have from an environmental perspective, which is important. For colocation actors, this can become a new way to compete. (ML) • Low temperature heat is better for local recycling; it is the district heating companies that have put themselves in high temperature situations. (ML) • There are examples of energy companies, not being transparent enough in pricing discussions. Some don’t even present proposal of their own. (ML) • There always needs be some backup to district cooling or district heating. However, many facilities have district cooling today. (ML) • Large internet giants such as Facebook are often so focused on their methodology for building data centers that they do not even want to discuss heat recovery. (MF) • Many of the northern regions in Sweden, which have been active in the data center development from the start, are years ahead regarding experience and knowledge of data center establishment. They know how to attract and satisfy investors who want to build data centers. It will be difficult for new regions to compete against these experienced regions. (MF) • Measurement of delivered heat can be difficult. It is important to decide who will carry out the measurements and how to control that the measurement is correct. (PN) C.2.5.11 Competition to District Heating from Other Cooling Options • Free cooling will be used by all who dispose of the option. That trend is not in decline as there are major advantages. The only way to break the free cooling trend is to regulate heat disposal. (HH) • Geothermal heating and cooling storages is are tough competitors to district heating and cooling, when it comes to new property developments in cities. (MG) C.2.5.11.1 Direct Local Recycling • If it is possible to recycle 28 degrees, then it is perfect for heating residences. (HH) • From an image perspective, we wanted to recover the heat as local as possible. Then there was a business case for both property owner and for us; we achieved a lower cost of cooling and the property owner achieved a profit from heating the property. (LG) C.2.5.11.2 Absorption Cooling and Heat Powered Cooling • A new trend for data centers in Europe is absorption cooling, possible when the cooling system delivers 50 oC. (GS) • For heat powered cooling, we need to add approximately 6 MW of heat to produce 5 MW of cooling. When regarded from an energy efficiency perspective, it is more expensive than an electricity powered heat pump, but the electricity is more expensive than the heat we use. (MG) • An absorption cooling machine powered by hot water has a hard time competing with a modern facility for free cooling in a data center. An absorption cooling machine is very expensive to establish and has low energy efficiency. During the warmer part of the year it will always be cheaper to produce cooling with an adiabatic refrigerant cooler or a cooling tower. There are only a few hours per year when it is not possible to operate an adiabatic refrigerant cooler. (JL)

xxxvi www.kth.se