Sampling Based on Sobol Sequences for Monte Carlo Techniques

Home , Latin hypercube sampling

Proceedings of Building Simulation 2011: 12th Conference of International Building Performance Simulation Association, Sydney, 14-16 November.

SAMPLING BASED ON SOBOL0 SEQUENCES FOR MONTE CARLO TECHNIQUES APPLIED TO BUILDING SIMULATIONS

Sebastian Burhenne1,∗, Dirk Jacob1, and Gregor P. Henze2 1Fraunhofer Institute for Solar Energy Systems, Freiburg, Germany 2University of Colorado, Boulder, USA ∗Corresponding author. E-mail address: [email protected]

ABSTRACT numerically expensive when the number of analyzed Monte Carlo (MC) techniques are commonly used to parameters (k) is large as its volume increases dramat- perform uncertainty and sensitivity analyses. A key el- ically with k. ement of MC methods is the sampling of input param- In this paper different sampling techniques are ana- eters for the simulation, where the goal is to explore lyzed with respect to the estimator of the mean of the the entire input space with a reasonable sample size result and how quick this estimator converges to the (N). The sample size determines the computational true mean (i.e., expected value) with respect to the cost of the analysis since N is equal to the required sample size. Another analyzed measure of the perfor- number of simulation runs. Quasi-random (QR) se- mance of the sampling strategy is its robustness. Ro- quences such as the Sobol0 sequences are designed to bustness can be measured via the standard error of the generate a sample that is uniformly distributed over estimated mean. This is done using multiple MC sim- the unit hypercube. In this paper, sampling based ulations and analyzing their results. A way to visualize on Sobol0 sequences is compared with other standard the robustness is to compare the empirical cumulated sampling procedures with respect to typical building density functions (CDFs) of several repetitions of the simulation applications. The work revealed that for MC simulation (Helton and Davis, 2003). the most of the analyzed aspects the sampling based Macdonald analyzed the performance of random sam- on Sobol0 sequences performs better than the other in- pling, stratified sampling and Latin hypercube sam- vestigated sampling techniques. pling applied to the evaluation of a building model INTRODUCTION (Macdonald, 2009). This paper extends his work by applying a sampling technique based on Sobol0 se- Due to the substantial influence of uncertain param- quences to a building simulation model. Furthermore, eters on building performance, uncertainty and sensi- models with different properties than the model used tivity analyses will become an important part of the by Macdonald are analyzed. As a test case, a simple building performance simulation and the design pro- mathematical model and a typical building simulation cess of low energy buildings. In an uncertainty analy- model are used. sis the modeler quantifies the uncertainty in the model output given the uncertainty in the model input. This SAMPLING TECHNIQUES goes often hand in hand with a sensitivity analysis Sampling is the process of exploring the domain of in- where the aim is to apportion the uncertainty in the terest (e.g., x1). That can be done randomly, where model output to the uncertainty in the model input the random numbers are independent realizations of a (Saltelli et al., 2008, pg. 1). Both analyses give in- random variable (Sobol’ and Levitan, 1999). In com- sights to the driving parameters or variables of the puter experiments, pseudo-random numbers or quasi- model and the model structure. random numbers are used. These numbers are gener- Several examples of Monte Carlo based uncertainty ated using an algorithm or a sequence of numbers that and sensitivity analyses applied to building sim- fulfill requirements as if they were true random num- ulations exist (Lomas and Eppel, 1992; de Wit, bers. The properties of the samples can be analyzed by 2003; Mara and Tarantola, 2008; Macdonald, 2009; the use of statistical tests (Sobol’ and Levitan, 1999). Burhenne et al., 2010a,b). Compared to sampling In the context of this paper the language and environ- methods already applied in various building perfor- ment R for statistical computing is used to generate the mance simulation applications (e.g., random sam- samples (R Development Core Team, 2010). pling, stratified sampling and Latin Hypercube sam- Random sampling pling) the sampling based on Solbol0 quasi-random sequences is expected to be more effective in exploring A random sample can be generated by a pseudo- the input parameter space. This space is a unit hy- random number generator which is available in many percube (Ω) with k dimensions. Exploring the unit software packages. A sample is randomly distributed hypercube with a sufficient sample density becomes in a defined interval according to some distribution

- 1816 - Proceedings of Building Simulation 2011: 12th Conference of International Building Performance Simulation Association, Sydney, 14-16 November.

(e.g., uniform distribution in the interval [0,1], hence Xi ∼ U(0, 1) with i = 1, 2, ..., N). For small sample sizes (N), the samples can contain clusters and gaps as 1.0 shown in Figure 1 on line a. Regions with gaps are not taken into account in the statistical analyses for any 0.8 uncertainty or sensitivity analysis and function values 0.6 in the regions with clusters are overemphasized in the 3 , random x 0.4 calculations (Saltelli et al., 2008, pg. 83). The sample 1.0 0.8 on line b was drawn using the same pseudo-random 2 , random 0.6 x 0.2 0.4 number generator but shows a better coverage of the 0.2 0.0 interval. 0.00.0 0.2 0.4 0.6 0.8 1.0

x1, random a Figure 2: Three-dimensional plot of the pseudo- b randomly sampled points in the parameter space x1, x2 and x3. The color of the points varies from red to black depending on the value of x2. That color 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 variation allows a better interpretation of the three- Figure 1: Two examples of sampling with a pseudo- dimensional plot. random number generator.

The unbiased mean and variance of the model output way. For this reason Figure 3 shows the variables can be calculated by the following equations (Saltelli x1, x2 and x3 plotted against each other in two- et al., 2008, pg. 59): dimensional plots. The plots for random sampling show clusters and gaps.

N Stratified sampling 1 X Y¯ = y (1) Figure 1 showed that a random sample may contain N i i=1 clusters and gaps. Using a stratified sampling tech- N nique can solve that problem. In a scheme which ap- 1 X ¯ 2 Var(Y ) = (yi − Y ) . (2) plies stratified sampling, the domain of x is divided N − 1 i i=1 into subintervals. Each of the subintervals contains the same number of sample points. These points are sam- The mean and the variance resulting from the sample pled randomly within each subinterval using a pseudo- and calculated with these two equations are uncertain. random number generator. If one compares Figure 1 Based on the central limit theorem, the uncertainty in with Figure 4 it is obvious that the stratified sampling the estimate of the mean can be quantified with the technique ensures the avoidance of clusters and gaps standard error at a certain resolution. r Var(Y ) SE(Y¯ ) = . (3) N a

This equation shows that the uncertainty decreases b slowly when N increases since it depends on the square root of N. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 In the following example a three-dimensional parameter space is used to illustrate the properties of the Figure 4: Two examples of stratified sampling. The sampling methods. However, the reader should keep position of the points within each subinterval is chosen in mind that exploring the parameter space becomes randomly. harder as the number of analyzed parameters (k) increases. The sampling is performed according to a uni- The mean and variance can be calculated in the same form distribution in the interval [0,1]. Figure 2 shows way as for pseudo-random sampling (see Equations 1 a three-dimensional plot of the parameter space x1, x2 and 2). and x3 with N = 128. The number was chosen be- In multivariate stratified sampling the same technique cause of the properties of the sampling based on Sobol0 is applied. Figure 5 shows a two-dimensional param- sequences which will be explained later. For pseudo- eter space with a stratified sampling with 10 strata for random sampling any N can be chosen but for the sake each parameter. That results in 100 cells where one of comparability N = 128 was used. point is in each cell. For a given resolution stratified With a three-dimensional plot it is a difficult task to sampling results in less uncertain mean and variance check if the parameter space is explored in a proper estimates than pseudo-random sampling (Saltelli et al.,

- 1817 - Proceedings of Building Simulation 2011: 12th Conference of International Building Performance Simulation Association, Sydney, 14-16 November. 1.0 1.0 1.0 0.8 0.8 0.8 0.6 0.6 0.6 0.4 0.4 0.4 2 , random 3 , random 3 , random x x x 0.2 0.2 0.2 0.0 0.0 0.0

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

x1, random x1, random x2, random 1.0 1.0 1.0 0.8 0.8 0.8 0.6 0.6 0.6 0.4 0.4 0.4 2 , stratified 3 , stratified 3 , stratified x x x 0.2 0.2 0.2 0.0 0.0 0.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

x1, stratified x1, stratified x2, stratified 1.0 1.0 1.0 0.8 0.8 0.8 0.6 0.6 0.6 2 , LHS 3 , LHS 3 , LHS 0.4 0.4 0.4 x x x 0.2 0.2 0.2 0.0 0.0 0.0

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

x1, LHS x1, LHS x2, LHS 1.0 1.0 1.0 0.8 0.8 0.8 0.6 0.6 0.6 2 , Sobol' 3 , Sobol' 3 , Sobol' 0.4 0.4 0.4 x x x 0.2 0.2 0.2 0.0 0.0 0.0

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

x1, Sobol' x1, Sobol' x2, Sobol' Figure 3: Three sampled parameters plotted against each other in pairs using different sampling techniques. Left to right: x1 vs. x2, x1 vs. x3, and x2 vs. x3. Top to bottom: Pseudo-random, stratiﬁed sampling, Latin hypercube sampling and sampling based on Sobol 0 sequences.

2008, pg. 80). Note that the required sample size for hypercube sampling with two parameters, 10 intervals doing this design is and a sample size (N) of 10 is shown in Figure 6. In Figure 6 the unique property of the sampled points of a N = mk. (4) LHS is easily visible: each sampled point is associated with one of the 10 rows and one of the 10 columns. Hence, for a stratified sampling with 10 strata (m) and In Figure 3 the Latin hypercube sampling shows gaps 5 parameters (k) a sample size (N) of 105 = 100, 000 and clusters like the random sampling because the is required. sample size is too small to generate a sample with the In Figure 3 the stratified sampling shows gaps and same density across the parameter space. clusters like the random sampling. The reason is that The mean and variance can be calculated in the same a sample size of 125 (128 like for the other techniques way as for pseudo-random sampling (see Equations 1 could not be used because of the property described by and 2). The LH sampling was implemented using the Equation 4) leads to 5 strata which is not sufficient to R package lhs (Carnell, 2009). avoid visible clusters and gaps. Sampling based on Sobol0 sequences Latin hypercube sampling The investigated method is based on Sobol0 se- Latin hypercube sampling (LHS) is a particular kind quences. Sobol0 sequences belong to the family of of stratified sampling. One feature is that each param- quasi-random sequences which are designed to gen- eter is stratified over s > 2 intervals (levels) where erate samples of multiple parameters as uniformly as the same number of points are located in each interval possible over the multi-dimensional parameter space (Saltelli et al., 2008, pg. 76). An example of a Latin (Saltelli et al., 2010). The biggest difference to

- 1818 - Proceedings of Building Simulation 2011: 12th Conference of International Building Performance Simulation Association, Sydney, 14-16 November.

tributed than the points produced with the other sam- 1.0 pling techniques. As a result, the discrepancy in the exploration of the multi-dimensional parameter space 0.8 is lower compared to the other sampling techniques. The mean and variance can be calculated in the same 0.6 way as for the other sampling techniques (see Equa- tions 1 and 2). 2 , stratified x 0.4 One result of the low discrepancy is that the estimated mean of a function Y(x1, x2, ..., xk) evaluated at the

0.2 points of a sampled input matrix

0.0  (1) (1) (1)  x1 x2 ··· xk 0.0 0.2 0.4 0.6 0.8 1.0  x(2) x(2) ··· x(2)   1 2 k   . . . .  x1, stratified  . . .. .  Min =   Figure 5: Scatterplot of a two-dimensional stratiﬁed  (N−1) (N−1) (N−1)   x1 x2 ··· xk  sampling. The position of the points within each cell  (N) (N) (N)   x x ··· x  is chosen randomly. 1 2 k

will converge quicker to the true mean than in the case

1.0 of pseudo-random sampling (Saltelli et al., 2008, pg. 83). How quick depends on the model structure and 0

0.8 will be analyzed later. The properties of the Sobol sequences require a sample size of 0.6 N = 2j (5) 2 , LHS x

0.4 where j ∈ N+. The sampling based on Sobol0 sequences was implemented using the R package rand-

0.2 toolbox. For the repetitions of the MC simulation the sampling was done using the Owen type of scrambling with a random seed (Dutang, 2009). 0.0

0.0 0.2 0.4 0.6 0.8 1.0 TEST MODELS Simple mathematical model x1, LHS

Figure 6: Scatterplot of a LHS: x1 vs. x2. The col- The ﬁrst model used in this paper is a simple math- ored dotted lines indicate to which intervals the sam- ematical model. It has the advantage that analytical pled point belongs. For illustration reasons this is just solutions are available which leads to a straight for- plotted for two points. However, each point has that ward analysis. The model was introduced by Stefano property. Tarantola (Tarantola, 2010). The 2-dimensional model equation is 2 f(x1, x2) = 4x + 3x2. (6) pseudo-random numbers is that the sample values are 1 chosen under consideration of the previously sampled with the input distributions points and thus avoiding the occurrence of clusters and gaps. 1 1 x1, x2 ∼ U(− , ). One criterion for assessing the performance of a good 2 2 sampling method is the discrepancy in the exploration Figure 7 shows a contour plot of the model. of the multi-dimensional parameter space. The dis- The expected value (E(f(x1, x2))) is 0 crepancy metric was deﬁned by Ilya M. Sobol and is Z 0.5 Z 0.5 2 the maximum deviation between the theoretical den- E(f(x1, x2)) = (4x1 + 3x2)dx1dx2 −0.5 −0.5 sity dt = 1/N and the point density di in an arbitrary 1 hyper-parallelepiped (Pi) within the parameter space = . (7) (hypercube) (Saltelli et al., 2010). The sampling based 3 on Sobol0 sequences is designed to generate samples The availability of the analytical solution makes it pos- with low discrepancy. sible to compare the mean of the function which is an Figure 3 shows that the points produced by a sam- estimator of the expected value to the true expected pling based on Sobol0 sequences are more evenly dis- value.

- 1819 - Proceedings of Building Simulation 2011: 12th Conference of International Building Performance Simulation Association, Sydney, 14-16 November.

were executed to perform the analysis (100 repetitions for 4 techniques with a sample size of 256 for each 0.4 MC simulation). The simulations were performed in parallel on a Linux-based computer cluster with 96 0.2 processor cores. However, in order to reduce the com-

2 puting time, it is desirable to use an appropriate sim- x 0.0 ple model for thermal building simulation. As a zone model the simple hourly method (SHM) according to

-0.2 the ISO 13790 standard is used (ISO 13790, 2008). This zone model is based on ﬁve resistances and one

-0.4 capacity. The model was calibrated for this building; it showed a good agreement with the measured room -0.4 -0.2 0.0 0.2 0.4 temperature and the heating demand of the building (Burhenne and Jacob, 2008). The object-oriented and x1 equation-based modeling language Modelica is used Figure 7: Contour plot of the simple mathematical to describe the system (Elmqvist, 1997) and the simu- model. lations are conducted using the software Dymola 2012 (Dassault Systemes` AB, 2011). Building simulation model In actuality, the building is an office building heated by a gas boiler. For this analysis, however, it is assumed The building simulated is a typical German building that it is a residential building with 12 occupants. A with a net floor area of 436 m2. The model was solar thermal collector with 25 m2 area and a 2,000 introduced in (Burhenne and Jacob, 2008; Burhenne liter storage tank are modeled. The collector model et al., 2010b). There is no air-conditioning available is implemented in Modelica using a plug flow model in the building and the heat is emitted by radiators description (Isakson and Eriksson, 1994). The collec- equipped with thermostatic control valves. The build- tor flow rate is controlled by an on/off controller; the ing is equipped with sensors (outside temperature, heat storage tank is modeled as a simple one resistor / one meter, room temperatures etc.) to allow for a valida- capacitor (R1-C1) network. The radiation processor tion of the simulation. The main building parameters is implemented according to an equation-based model are shown in Table 1 and Figure 8 is a 3D-plan of the (written in the modeling language Neutral Model For- building. mat; (Sahlin, 1996)) from the simulation software IDA Table 1: Building parameters. ICE (Sahlin et al., 2004). The solar thermal system is designed for domestic hot water and space heating. parameter value unit When the heat from this solar thermal system is not 2 sufficient, a gas boiler meets the remainder of the load A (area to volume ratio) 0.38 m V m3 of the building. Further details on the model can be W found in (Burhenne et al., 2010b). The primary model U-value (mean U-value) 0.53 m2K result is the annual solar fraction of the solar thermal 2 Awin (total window area) 106 m system. It is assumed that the mass flow rate of the domestic hot water (m˙ ) and the air change rate (ACH) are uncertain. Furthermore, the number of people (occ) present at a particular time and the set point for the room temperature (set) cannot be determined exactly. Therefore, these four values (m˙ , ACH, occ, set) are varied in the MC simulation. The schedule for the domestic hot water flow rates is generated with a program which was developed in the Solar Heating and Cool- ing Program of the International Energy Agency (IEA- SHC), Task 26: Solar Combisystems (Jordan and Va- Figure 8: 3D-plan of the building. jen, 2003). The air change rates, the number of occupants and the room temperature set point are also Monte Carlo (MC) simulations require many simula- implemented using a schedule. The variation of the tion runs and are therefore computationally expensive. schedule values is implemented by multiplying a sam- Especially the tests with repetitions of the analysis re- pled scaling factor or adding a scaling summand. The quire many simulations but are necessary to evaluate scaling parameters are listed in Table 2. Note that the the performance of the different sampling techniques. values for the scaling summands for the number of oc- In this paper more than 100,000 one-year simulations cupants and the set point are rounded (integers for the

- 1820 - Proceedings of Building Simulation 2011: 12th Conference of International Building Performance Simulation Association, Sydney, 14-16 November. occupancy and values with one fractional digit for the Building simulation model set point). Figure 11 compares the convergence to the mean (solar fraction) for the different sampling strategies for Table 2: Distribution parameters. the building simulation model. The black horizontal parameter distribution µ σ line is the mean value after a MC simulation with random sampling and a sample size of 25,600. Due to this m˙ scaling factor normal 1 0.1 large sample size this value can be taken as a refer- ACH scaling factor normal 1 0.2 ence. It can be seen that the sampling based on Sobol0 occ scaling summand normal 0 2 sequences and the LH sampling converge very quick set scaling summand normal 0 1 and produce reasonable results for very small sample sizes. However, at a sample size of 256 all sampling strategies converged and produce comparable results. RESULT ANALYSIS AND DISCUSSION

Simple mathematical model 0.23 Random sampling Stratified sampling Figure 9 shows a comparison of the different sampling Latin hypercube sampling techniques used to compute the mean of the function 0.22 Sampling based on Sobol' sequences with different sample sizes. The random sampling

shows the worst convergence to the analytical mean 0.21 of the function. However, the other three techniques already converged at a sample size of 64 and the Latin hypercube sampling shows the fastest convergence. 0.20 mean (solar fraction) 0.19

0.6 Random sampling Stratified sampling 0.18 Latin hypercube sampling 0.5 Sampling based on Sobol' sequences 0 100 200 300 400 500 600

sample size ) 0.4 2

, x analytical mean 1

x Figure 11: Comparison of the different sampling tech- ( f 0.3 ( niques applied to the evaluation of the building simulation model. mean 0.2 Figure 12 shows the comparison of the estimated

0.1 CDFs for the building simulation model. The model evaluation was repeated 100 times for each sampling

0.0 technique. The sampling based on Sobol0 sequences 0 100 200 300 400 500 has the least variation followed by the Latin hypercube and the stratiﬁed sampling. The CDFs constructed for sample size the random sampling show the most variability. Figure 9: Comparison of the different sampling tech- CONCLUSION niques applied to the evaluation of the simple mathematical model. Different sampling strategies were analyzed with respect to the convergence of the mean estimate to the true expected value and their robustness. The perfor- In this paper, cumulative distribution functions (CDFs) mance of the techniques depends on the number of in- are used to visualize the robustness (i.e., stability) put parameters (k) and the properties of the analyzed of results obtained by different sampling strategies. model (e.g., nonlinearity). The Latin hypercube sam- The sampling and model evaluation was repeated 100 pling and the sampling based on Sobol0 sequences had times for each sampling technique and the sample size the fastest convergence of the mean estimates. More- of 256. Figure 10 shows the comparison of the esti- over, the comparisons of the estimated CDFs showed mated CDFs for the simple mathematical model. One that the sampling based on Sobol0 sequences has the can see that in this case the robustness of the stratiﬁed least variations in the CDFs. Having less variations sampling and the sampling based on Sobol0 sequences proves that this sampling technique produces the most is the best followed by the Latin hypercube sampling. robust results. It is surmised that the higher the number The CDFs constructed for the random sampling show of analyzed input parameters, the better the sampling the most variability (i.e., the estimated mean produced based on Sobol0 sequences performs in comparison by random sampling is most uncertain). to the other sampling strategies. It is recommended

- 1821 - Proceedings of Building Simulation 2011: 12th Conference of International Building Performance Simulation Association, Sydney, 14-16 November.

Figure 10: Comparison of estimated CDFs for the simple mathematical model with 100 repetitions. to use either Latin hypercube sampling or sampling Dassault Systemes` AB 2011. Dymola. Dynamic Mod- based on Sobol0 sequences when Monte Carlo tech- eling Laboratory. Dymola Release notes. niques are applied to building performance simula- de Wit, S. 2003. Uncertainty in building simulation. tions and the sample size is limited because of com- In Malkawi, A. M. and Augenbroe, G., editors, Ad- putationally expensive models. vanced Building Simulation, pages 25–59. Taylor & ACKNOWLEDGEMENT Francis, Abingdon, UK. This study was funded by the Reiner Lemoine Dutang, C. 2009. randtoolbox: Generating and Test- Stiftung. ing Random Numbers. REFERENCES Elmqvist, H. 1997. Modelica - A Uniﬁed Object- Burhenne, S., Elci, M., Jacob, D., Neumann, C., and Oriented Language for Physical Systems Modeling. Herkel, S. 2010a. Sensitivity analysis with building Simulation Practice and Theory, 5(6). simulations to support the commissioning process. In ICEBO 2010, 10th International Conference for Helton, J. and Davis, F. 2003. Latin hypercube sam- Enhanced Building Operations, Kuwait City. pling and the propagation of uncertainty in analyses of complex systems. Reliability Engineering & Sys- Burhenne, S. and Jacob, D. 2008. Simulation models tem Safety, 81(1):23–69. to optimize the energy consumption of buildings. In ICEBO 2008, 8th International Conference for En- Isakson, P. and Eriksson, L. O. 1994. MFC 1.0β. hanced Building Operations. Matched Flow Collector Model for simulation and testing. User’s manual. Technical report, IEA Burhenne, S., Jacob, D., and Henze, G. P. 2010b. Un- SH&CP Task 14. Royal Institute of Technology certainty analysis in building simulation with Monte (KTH), Stockholm. Carlo techniques. In SimBuild 2010, 4th National ISO 13790 2008. Energy performance of buildings Conference of IBPSA-USA, New York City. - Calculation of energy use for space heating and Carnell, R. 2009. lhs: Latin Hypercube Samples. cooling.

- 1822 - Proceedings of Building Simulation 2011: 12th Conference of International Building Performance Simulation Association, Sydney, 14-16 November.

Figure 12: Comparison of estimated CDFs for the building simulation model with 100 repetitions.

Jordan, U. and Vajen, K. 2003. Handbuch. DHW- Sahlin, P., Eriksson, L., Grozman, P., Johnsson, H., calc. Werkzeug zur Generierung von Trinkwasser- Shapovalov, A., and Vuolle, M. 2004. Whole- Zapfproﬁlen auf statistischer Basis. Version 1.10 building simulation with symbolic DAE equations (2003). Technical report, Technical University of and general purpose solvers. Building and Environ- Denmark / Universitat¨ Kassel, Lyngby / Kassel. ment, 39(8):949–958. Lomas, K. and Eppel, H. 1992. Sensitivity analy- Saltelli, A., Annoni, P., Azzini, I., Campolongo, F., sis techniques for building thermal simulation pro- Ratto, M., and Tarantola, S. 2010. Variance based grams. Energy and Buildings, 19(1):21–44. sensitivity analysis of model output. Design and estimator for the total sensitivity index. Computer Macdonald, I. 2009. Comparison of sampling tech- Physics Communications, 181(2):259–270. niques on the performance of Monte-Carlo based sensitivity analysis. In 11th International IBPSA Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Conference, pages 992–999, Glasgow. Cariboni, J., Gatelli, D., Saisana, M., and Tarantola, S. 2008. Global Sensitivity Analysis. The Primer. Mara, T. A. and Tarantola, S. 2008. Application John Wiley & Sons, Ltd. of global sensitivity analysis of model output to building thermal simulations. Building Simulation, Sobol’, I. M. and Levitan, Y. L. 1999. A pseudo- 1(4):290–302. random number generator for personal computers. Computers & Mathematics with Applications, 37(4- R Development Core Team 2010. R: A Language and 5):33–40. Environment for Statistical Computing. R Founda- Tarantola, S. 2010. Variance based Sensitivity Analy- tion for Statistical Computing, Vienna, Austria. sis. In The Sixth Summer-school on Sensitivity Anal- Sahlin, P. 1996. NMF HANDBOOK. An Introduc- ysis, Florence. tion to the Neutral Model Format. NMF version 3.02. Technical report, Royal Institute of Technol- ogy (KTH).

- 1823 -