Automation in Construction 130 (2021) 103849

Contents lists available at ScienceDirect

Automation in Construction

journal homepage: www.elsevier.com/locate/autcon

An evidence of cognitive benefits from immersive design review: Comparing three-dimensional perception and presence between immersive and non-immersive virtual environments

Daniel Paes a,*, Javier Irizarry b, Diego Pujoni c a School of Built Environment, Massey University, Auckland, New Zealand b School of Building Construction, Georgia Institute of Technology, Atlanta, GA, USA c Institute of Biological Sciences, Federal University of Minas Gerais, Belo Horizonte, MG, Brazil

ARTICLE INFO ABSTRACT

Keywords: The adoption of VR solutions in design review has grown rapidly in the construction sector worldwide. However, Virtual reality the extent to which these enhance professionals' cognitive capabilities in the task – an essential step in deter­ Immersive environments mining their validity and effectiveness – is still unclear. Through a controlled experiment, this study sought to Design technology quantitatively verify the ability of an immersive VR system to provide users with an enhanced 3D perception of Human-computer interaction the virtual model and greater levels of presence compared to a non-immersive one. Controlling for individual Three-dimensional perception Presence factors and order effects, findings indicate that the immersive system improves the 3D perception of the model Human factors and provides more immersive experiences, which is expected to benefitcollaborative design review and increase Experimental design productivity.

1. Introduction participate in the design process. Design review meetings – also referred to as coordination meetings – take place at different stages of the design Advanced visualization solutions such as Immersive Virtual Reality process, during which the participants get together to communicate, (IVR) have been developed and implemented to improve decision- evaluate, merge, and generate design solutions [92] with the support of making and problem-solving in collaborative design review and con­ design representations. However, seeking input from all stakeholders is structability analysis meetings [3,8,18,59,62], in construction safety difficult if they cannot adequately interpret the design representation [73,91] and disaster evacuation training [54], in the prediction of [19]. Therefore, an underlying condition in the design review activity is human-building interactions [1,7,31,32,34,42], as well as in construc­ to achieve a shared understanding of the architectural design. In this tion and architecture education [12,55,78,79]. direction, the representation format utilized to convey the envisioned Steuer [81] strategically defined VR (Virtual Reality) as a kind of building can largely affect how people understand that information [13] presence experience, allowing for the distinction among different VR and the overall quality of decision-making and problem-solving [3]. An systems according to the level of presence provided. In turn, Wann and adequate representation would communicate the designer's intentions Mon-Williams [88] stated that a system must satisfy criteria that arise in a less cognitively demanding format, allowing for a more effective from human perception to be considered a Virtual Environment (VE). and smooth review process [12]. The usefulness of a representation Both definitions do not rely on a system's technological apparatus and depends on how suitable it is for its purposes [60]; thus, the adoption of appear better aligned with the general purpose of virtual environments design representations that are closer to the existential-spatial human for the built environment, i.e., to deliver lifelike simulations. In the experience in the real world would contribute to the development of context of design review, VR could be defined as the experience of buildings that match end-users demands more effectively, whether presence in a fictitious architectural environment through its represen­ technical, functional, or symbolic ones [23]. Many scholars argue that tation, whichever the type of representation in use (not necessarily 2D (two-dimensional) drawings are limited in their usefulness and three-dimensional digital models). ability to convey architecture [38], requiring additional cognitive effort Typically, many stakeholders and experts from various disciplines to visualise the object represented, especially for complex structures

* Corresponding author. E-mail addresses: [email protected] (D. Paes), [email protected] (J. Irizarry), [email protected] (D. Pujoni). https://doi.org/10.1016/j.autcon.2021.103849 Received 16 February 2021; Received in revised form 1 June 2021; Accepted 21 July 2021 0926-5805/© 2021 Elsevier B.V. All rights reserved. D. Paes et al. Automation in Construction 130 (2021) 103849

[43]. This additional effort is precisely the need to extrapolate the technology for collaborative design review is to understand whether and drawing's scale to one's internal scale, requiring significanttraining. 2D to what extent it enhances users' ability in obtaining and understanding drawings may not deliver a good sense of size and volume of space, the spatial relationships depicted, or yet, their three-dimensional restricting stakeholders' ability to understand and suggest necessary perception and sense of presence [80,88]. Methods to verify spatial alterations to a design [30]. perception and presence in virtual environments are difficultto develop In contrast, the rich, interactive, and intuitive nature of 3D (three- since variables of interest often result from complex and eventually dimensional) models can greatly enhance the participants' understand­ unclear cognitive phenomena. Nonetheless, identifying and character­ ing of the design under revision [19]. Moreover, advanced visualization izing technological and human factors (VR system's properties and user- solutions such as IVR are expected to facilitate the communication and related factors) that affect presence and perception in virtual environ­ shared understanding of a project's 3D model [56,85,92], which ments has been acknowledged by many scholars as one of the most contribute to addressing issues of constructability [8], operability and critical steps for the development of VR systems to enhance human ca­ maintainability of facilities at early design stages towards minimizing pabilities in various contexts [40,76,77,96]. As suggested by Wann and construction, operation and maintenance costs associated with flaws Mon-Williams [88] and Chandrasegaran et al. [13], virtual environ­ arising from poor solutions [2,4]. In sum, the idea that IVR systems ments should be designed around human perceptual capabilities in the deliver better design representations leads to the expectation that they context of the task to be performed. Thus, only by understanding would also facilitate problem-solving in the design process and increase whether, to what extent, in what circumstances, and to whom virtual productivity. environments are beneficialit will be possible to definethe ergonomic, VR simulations have become a popular technology for design review environmental, technological and representational parameters upon due to perceived benefits associated with the representation of scale, which such systems would be actually efficient and increase produc­ depth, and volume [12]. These can be displayed within Immersive En­ tivity of the construction sector. vironments (IE) – a synonym for IVR technology – where the user can In this context, the general research hypothesis is that an IVR system literally walk through and interact with the virtual content [16]. VR would enhance users' three-dimensional perception of a BIM model and systems can be of different types. Generally, they comprise four main sense of presence in the virtual environment compared to a conventional components: a) a computer-generated 3D model (the simulation), b) a non-immersive VR system. The research questions and respective display, c) interaction/navigation devices to interact with the virtual alternative statistical hypotheses (H1) are as follows: model (controllers, data gloves, keyboard, etc.), and d) a software application that orchestrates all the different components. There are 1) Does the IVR system enhance users' three-dimensional perception of different types of display systems as well, either monoscopic or stereo­ a BIM model? H11: There is a difference in 3D perception (dependent scopic ones, such as mobile displays (smartphones and tablets), com­ variable) between VR modes (independent variable). puter monitors, head-mounted displays (HMD), binocular omni- 2) Does the IVR system enhance users' sense of presence in a BIM-based orientation monitors (BOOM), projection-based panoramic displays, virtual environment? H21: There is a difference in presence cave automatic virtual environments (CAVE™), 3D glasses (coupled (dependent variable) between VR modes (independent variable). with computer monitors, projection-based, and CAVE-like displays), and virtual retinal displays (VRD). VR platforms can be categorized into non- More specifically, the experiment aims at finding out whether and immersive and immersive. Non-immersive, low-end VR systems display how changes in a single independent variable (VR mode) of two monoscopic perspective views of a digital model. Interaction is usually different values, namely, the niVR (non-immersive VR: a BIM model limited to navigation through the environment using a mouse and displayed through a laptop screen) and the IVR (immersive VR: a BIM keyboard, which allow movement forward, backward, left and right. In model displayed through a commercial head-mounted display) systems, high-end immersive systems (IVR), tracking devices detect a user's head induce changes in the dependent variables, namely, 3D perception and and body movements and the display device projects stereoscopic im­ presence, controlling for ten different user-related confounding vari­ ages of a digital model [7]. ables, that is, individual characteristics of age, gender, educational level, bachelor's major, current major, experience in design review, computer 2. Problem statement and research questions usage, experience with 3D virtual environments, familiarity with the experiment environment, and spatial ability. Measures of 3D perception Studies on virtual modeling and advanced visualization in the built and presence in each experimental condition are used to perform a environment often overlook the cognitive processes that underlie the direct comparison between niVR and IVR systems. understanding of 3D representations, focusing on the analysis and description of visualization systems' properties instead. Previous 3. Research background research on the topic – mostly qualitative, exploratory and case studies – explored VR implications on productivity, task performance time and This section provides the theoretical background and rationale for other task-related effectiveness indicators [45], neglecting the cognitive the two research questions mentioned in the previous one. benefits that the technology may provide – an issue that precedes pro­ ductivity – and disregarding the human factors inherent to VR experi­ 3.1. Relationship between VR systems and 3D perception (research ences [65]. Seeing that the most widely accepted definitionof VR – one's question 1) experience of presence in virtual worlds [81] – places the user as its central, defining component [80], any analysis of the effectiveness of In the design of specialized facilities and increasingly complex virtual environments should be conducted on measures of user experi­ buildings, designers and end-users are frequently required to assess if ence/performance, in conformity with main references in the fields of dimensions and proportions of a space are adequate for its purposes – human-computer interaction (HCI) and cognitive psychology. As per whether it is a hospital surgery room, a religious temple, or a circulation Higuera-Trujillo et al. [35], research in the fieldis limited in three major area. Whenever the focus is to design spaces for optimal use, VR simu­ aspects: a) obsolescence of the studied VR platform, which stresses the lations can aid in the determination of layout configurations,room areas importance of critically and comparatively updating their validity, b) and proportions, sizes of walls and openings, and height and slope of disregard of cognitive aspects of user experience that underlie human ceilings in regards to the activities to be undertaken in the future space behavior and psychological state in virtual environments, and c) lack of [5,18,50,74]. A client may want to “feel” how spacious a sitting room is, adoption of user's objective responses in validation studies. or check how high the ceilings of a foyer or train station look; a land­ Thus, a fundamental step for IVR to be considered an effective scape architect may check if there are any visual obstructions to a

2 D. Paes et al. Automation in Construction 130 (2021) 103849 landmark of interest from a certain vantage point; nurses and surgeons The expectation is that depth perception would be enhanced by IVR may verify and provide feedback on the accessibility to key equipment technology's stereoscopic visualization feature, meaning more realistic in a surgery room; a facility manager may want to check whether there representations. Essentially, IVR aims to allow people to visually would be enough room to perform a certain repairing procedure on the perceive a virtual world as they perceive physical reality [7]. Interest­ HVAC machinery (some of these can be found in ingly, the higher realism of IVR stereoscopic representations (in terms of [2,51,53,57,58,87,93]). depth perception) is still an assumption and has not been tested to date. Often taken for granted, the feeling of the scale of space is, in reality, Accurate depth perception approximates the representation to the sustained by unconscious and dynamic processes of spatial perception. future artifact being represented. That is, whether a depictive repre­ While the processes of visual perception of 2D images are relatively well- sentation is able to describe and convey the location of points in space in known (e.g., [48,60,69]), the ones involved in the perception of dy­ a more accurate way – accuracy as being the degree of resemblance to namic 3D environments – such as the real world and VR simulations – the location of points in the physical world – the mental representation are still unclear [80]. Contemporary scholars may refer to the latter as created upon its perception will be closer to the physical or envisioned perception of layout or layout perception, which encompasses the idea reality, space, or object it aimed to describe originally [65]. In sum, the of perceiving the arrangement and displacement of objects in space. The level of (spatial) realism of representations is given by the accuracy of general argument is that people do not perceive space but objects in depth perception in relation to perception in the real world. Thus, the space, which invalidates the expression “spatial perception” [17,25], underlying question resides on whether our visual perceptions of the although adopted by many scholars (e.g., [30,40]). Alternatively, re­ real world are similar to our perceptions of a virtual world mapped from searchers have utilized a variety of terms to refer to the process of it (Fig. 3). In other words, are our perceptions of a virtual environment perception of the spatial configuration and three-dimensional arrange­ similar to our perceptions of a physical environment? The accuracy of ment of environments such as spatial understanding [74], spatial this mapping process gives the compatibility between perceptions of cognition [46], 3D visualization [36,37], or yet, spatial comprehension physical and virtual realities and establishes the visual realism of virtual [96]. environments [7]. Although the relevance and purposes of spatial perception in design In this context, the most important visual cues to reproduce the review may vary across disciplines and their specificinformation needs, three-dimensionality of the world might be the ones that allow for depth shared understanding of the general spatial relationships of a 3D model perception, such as occlusion, stereopsis and perspective. Consequently, by all stakeholders – not only experienced designers – is critical to these cues should be included in a virtual environment if its purpose is to achieving more integrated solutions [18,22]. Only by understanding the deliver accurate depth perception. This is particularly true for stereopsis, architectural representation, designers, construction agents, clients, and which is expected to greatly affect depth judgments [20]. end-users would be able to adequately evaluate a design and provide The possibility of depth perception through stereoscopic visualiza­ feedback on the implications of solutions developed. tion is a definingfactor of immersive VR systems and represented a real In this direction, great realism levels are expected to benefit the breakthrough for VR technology [7]. Nonetheless, little is known about understanding of representations and facilitate design review (Fig. 1). In whether stereoscopic visualization actually benefits the perception of turn, the level of realism of representations is largely affected by the depth from 3D models. Due to their direct association, distance esti­ extent to which these are able to convey depth. mation has been widely used as a metrics of depth perception in virtual Naturally, depth representation per se does not guarantee an accu­ environments (e.g., [27,39,71,75,84,89]). Therefore, question 1 aims to rate perception of a space's three-dimensional structure. One must verify if an IVR system would allow users to estimate distances more perceive depth from its representation. In other words, a given repre­ accurately, i.e., to have a more accurate three-dimensional perception of sentation can only be deemed more realistic if one actually perceives an architectural design compared to non-immersive VR technology. In three-dimensionality more accurately from it. Therefore, in reality, other words, three-dimensional perception is operationalized in this there is something in between depth representation and realism of study as one's understanding of the dimensions, proportions, and scale of representation shown in Fig. 1, and that is precisely depth perception or an architectural design [96], provided by estimates of egocentric dis­ three-dimensional perception (Fig. 2): a specific visual process within tances to objects in space and distances between objects in space (hor­ visual perception that accounts for the perception of depth of objects in izontal, vertical, and depth judgments combined). To avoid any space, and the dependent variable in hypothesis H1. misconceptions of nomenclature, this study adopts the term three- In constructing three-dimensional scenes and objects, the visual dimensional perception or simply 3D perception, as did Wann and system uses multiple sources of information or “visual cues”. The rela­ Mon-Williams [88] and Norcia and Gerhard [61], to refer to a particular tive importance of visual cues is still unknown, but they certainly perceptual process within visual perception that governs the interpre­ interact, reinforce, conflict with each other and build on one another. tation of visual sources of information about the three-dimensional Environments are typically rich in visual cues, which can be extremely structure of a space, that is, its three-dimensionality (X/horizontal, Y/ varied, with certain kinds of information present in some situations but vertical, and Z/depth dimensions). In sum, question 1 compares the not in others. For instance, at different distances, some sources fail to ability of two distinct VR systems (non-immersive vs. immersive) in deliver an adequate quality of information (they may vary or fade out conveying the three-dimensionality of the space represented. with distance) so that the visual system is forced to rely on others [17]. In turn, depth perception is the result of one's interpretation of depth 3.2. Relationship between VR systems and presence (research question 2) cues. These can be broadly categorized as either geometric (e.g., dis­ tance, direction) or featural cues (e.g., color, shading, texture) [46], and Presence is also expected to benefit the design review process, as it further categorized into primary (stereopsis and parallax) and secondary has been shown to correlate significantly with visual information cues (motion parallax, perspective, occlusion, size, texture, color and acquisition, learning [63], task performance, and even quality of design shading, light and shadow, among others) [44,47]. solutions, although one may still perform well while experiencing low

Fig. 1. Relationship between depth representation and design review.

3 D. Paes et al. Automation in Construction 130 (2021) 103849

Fig. 2. Relationship between depth representation and realism of representation.

Fig. 3. Compatibility between perceptions of virtual and physical worlds. Adapted from [7]. levels of presence [21]. Acquisition of visual information (also known as different VR systems. Khashe et al. [42] conducted a between-subject visual search) is part of the visual perception process, which involves study to examine the effects of immersive and non-immersive VR plat­ directed attention orienting one's sight towards visual information forms on compliance with environmental requests, reading perfor­ sources and selective processing of available information [25,26]. In the mance, presence and motion sickness. They also analyzed the influence VR-assisted design review, greater levels of presence are expected to of some individual factors on the dependent variables, as well as the provide users with enhanced ability to search, locate and identify visual interaction between presence and other dependent variables within each information, improving the visual perception of their virtual surround­ condition. In a within-subject study, Castronovo et al. [12] investigated ings and facilitating the identification of design issues [33,41]. Fig. 4 the relationship between presence and VR systems in design review. summarizes the relationship between presence and design review. Presence was given by participants' self-reported experiences of realism of movement, “physical” presence, attention, and their sense of “being 4. Research method part of the virtual environment”, among others. They utilized different simulations in each VR condition (Revit-based vs. Unity3D-based sim­ The research method is based on previous research on visual ulations). Ozcelik and Becerik-Gerber [64] designed a within-subject perception and sense of presence in virtual environments that have study to investigate the relationship between presence and potential adopted presence or visual perception questionnaires to assess user influencing factors (comfort, satisfaction, number/type of interactions, experience (e.g., [11,21,33,41,66,68,72,86,90,96]). perceived temperature) and found a positive correlation between perceived thermal comfort and presence when analyzing data from all 4.1. Past experiments conditions combined. Higuera-Trujillo et al. [35] conducted a thorough between-subject experiment to compare the differences in psychologi­ 4.1.1. Measuring presence cal, physiological, and presence responses among different display for­ ◦ The major critics to Witmer and Singer's [90] Presence Questionnaire mats, including photographs, 360 panoramas, and a virtual (e.g., [21,76,86]) argue that while the authors acknowledge that pres­ environment. Responses from display modes were “standardized” over ence derives from perceptual processes (involvement and immersion), physical-world responses beforehand to simplify the comparisons their instrument may not measure the psychological state of presence among display types. They also adopted the SUS presence questionnaire but a person's subjective opinions about various properties of a VR [86]. Naturally, presence responses were not collected in the physical system. Indeed, questions like “How much did the visual aspects of the environment condition. Results indicated that VR offers the closest-to- environment involve you?” or “How natural was the mechanism which reality experience with respect to the user's physiological responses, controlled movement through the environment?” [90] appear to relate and that physiological and psychological responses correlate with the more to the characteristics of the system than to a user's cognitive sense of presence. experience. Nonetheless, presence and spatial perception questionnaires have been adapted from Witmer and Singer's instruments and used in 4.1.2. Measuring three-dimensional perception various studies in the built environment field (e.g., In general, due to their direct association, distance estimation has [11,33,41,42,66,72,96]). been widely used as a metrics of depth/three-dimensional perception in A few among those studies sought to compare presence levels across both physical and virtual environments (e.g., [27,39,71,75,84,89]). In

Fig. 4. Relationship between presence and design review.

4 D. Paes et al. Automation in Construction 130 (2021) 103849 studies that compare distance estimation between virtual and real-world These are known as random or confounding variables [70], or yet, settings, the accuracy of virtual distance estimation is usually given by covariables [52]. Participants could perform better in 3D perception and the percentage of the absolute/actual distance estimated by a partici­ report greater levels of presence due to those factors. In this study, those pant [95]. However, when a study compares distance estimation be­ consist of individual factors of age, gender, educational level, bachelor's tween different virtual systems, the accuracy of estimation is provided major, current major, experience in design review, computer usage, by the deviation of a participant's estimates in the virtual environment experience with 3D virtual environments, familiarity with the experi­ with respect to her/his estimates in the real world [35,68]. ment environment, and spatial ability. There are various ways to Distance estimation can be performed through different methods. exclude or control confounding variables. In a within-subject experi­ Most studies adopt egocentric judgment techniques, i.e., distance esti­ mental design, these variables are almost fully controlled because a mation from the observer at a fixed position to a target object. Alter­ participant's performance in a condition is compared against her/his natively, distance estimation can be based on the time it takes a own performance in the other condition such that individual factors are participant to walk towards target-objects, also referred to as “time-to-- equally impactful in both conditions. Regardless of the unlikelihood of walk” estimates [95]. impacts of confounders on the dependent variables due to the experi­ mental design, their effects will be verifiedin data analysis to check for the effectiveness of measures taken to control them with the experi­ 4.2. Experimental design mental design. The lobby of the Caddell Building on the Georgia Tech campus was This is an empirical, quantitative, relational study, although it carries selected as the experiment environment. The reference mode mentioned many characteristics of experimental research (such as an experiment), above (PhE) is the actual lobby. It was selected for having a moderate as most human-computer interaction studies. The method is formally level of architectural complexity (diversity and characteristics of defined as a controlled experiment utilizing survey questionnaires to architectural elements) and being a representative of typical contem­ collect user experiences. porary institutional architecture (in terms of construction technology This study adopts a within-group experimental design, also referred and architectural typology). The lobby space is the entrance hall of a to as “within-subject” design, where participants are exposed to all two-story building of approximately 1024 m2 of total area. It consists of experimental conditions. Data collection makes use of various survey a single large room of a roughly squared floorplan (approximately 9 by questionnaires. Data analysis consists of comparing the performances of 9 m) and a ceiling height of approximately 4 m. A red-colored metal the same participants under different conditions. Because 3D perception staircase with glass railings is located at one of the corners of the room. A and presence involve significant cognitive functions, individual differ­ large floor-to-ceilingglass panel separates the interior from the exterior ences are expected to largely affect the outcomes. Such individual dif­ and allows a great amount of natural light into the room. Interior fin­ ferences are better controlled in a within-subject design, which excludes ishing materials include polished concrete on the floor, white paint on the variance between subjects due to those differences in the comparison walls and plasterboard ceiling, dark grey paint on doors, and red (or of effects of different conditions, since each participant serves as her/his dark orange) paint on the exposed steel structure (Fig. 6). own equivalent in the comparison [52]. Consequently, the power to detect existing differences is usually much higher in within-subject de­ signs than in between-subject designs given the same sample size [14,83]. Nonetheless, within-subject experiments must be carefully designed to ensure that the benefits of such a design (smaller sample size, greater power, etc.) outweigh potential drawbacks (practice/ learning/carry-over, fatigue, and expectancy effects) [83]. In this study, the number and values of the independent variable (a single independent variable of two different values) create two different experimental treatments or conditions, namely, the niVR (a BIM model displayed through a laptop screen) and the IVR (a BIM model viewed through a commercial head-mounted display, i.e., an immersive virtual reality system). A participant's performance in 3D perception and level of presence (dependent variables) are collected in each condition. It should be noted that a user's performance in 3D perception is deter­ mined – or “standardized” [35] – against her/his performance in the physical environment (PhE). The experimental design structure is shown Fig. 6. Caddell Building lobby. Retrieved from https://www.re-thinkingthefut in Fig. 5. ure.com/wp-content/uploads/2020/08/A1438-BLDGS-Iconic-Projects-John-an The dependent variables of 3D perception and presence could be d-Joyce-Caddell-Building-Image-8.jpg. affected by several factors that studies of this nature cannot fully control.

Fig. 5. Experimental design structure.

5 D. Paes et al. Automation in Construction 130 (2021) 103849

The decision between using a high- or low-fidelity model in the VR a. the Demographic Questionnaire (DQ), used to collect demographic modes is guided by the general research motivation: to identify benefits data of participants, including information on individual character­ of immersive visualization of BIM models. As pointed out by Berg and istics that could possibly affect 3D perception and presence. Vance [6] and Paes and Irizarry [67], the relevance of pictorial realism b. an adapted version of the Spatial Ability Test (Revised PSVT:R), a well- in virtual environments is strictly a function of the questions being accepted multiple-choice test used to collect human spatial ability explored. For designers who concentrate on the fit,form, and function of [94], which is another individual trait that can impact users' per­ a space, a model's geometry must be accurate and representative of the formance in spatial tasks [49]. In a study conducted by Oren et al. design solutions with respect to scale, size, orientation, and position. In [63], participants completed a spatial ability test so the researchers this scenario, decision-makers are less interested in the simulation's could control for variance of learning performance due to spatial pictorial realism and more interested in whether the design fulfills the ability differences. If any, spatial ability is expected to have little technical specifications.Thus, using high-fidelityrenderings may not be impact on the dependent variables in this study. The assumption is a priority in this case. Indeed, low-fidelity BIM models are the most that even those who demonstrate low spatial ability can achieve widely used simulations in design review and across the construction great levels of presence [90] and perform well in spatial perception industry [19]. Featural spatial cues [46] should be kept the same across tasks [10]. VR modes to allow for adequate observation of display effects (stere­ c. the 3D Perception Questionnaire (3DPQ), used to collect participants' opsis, field of view, and interactivity effects). Keeping graphics quality three-dimensional perception (horizontal, vertical, and depth judg­ the same across VR conditions eliminates the effects due to differences of ments combined) of an architectural design in each VR mode (niVR graphics and automatically leaves the effects due to the visualization and IVR) as well as in the reference mode (PhE). The 3DPQ comprises technology “at their will.” Thus, both VR conditions feature: a) first- twelve objective questions that prompt the respondents to estimate person view (FPV) walkthrough mode of navigation, b) level of egocentric distances to objects in space (egocentric distance esti­ graphics quality equivalent to the quality currently provided by BIM mation) and distances between objects in space (interobject distance software applications, c) identical levels of graphics quality, and d) estimation) (Table 1). The questionnaire was based on previous identical user interfaces (UIs). Using the same converter application in studies that have developed and used similar instruments (e.g., the non-immersive VR condition ensures similar and equally free-of- [41,68,96]). In order to ensure its qualitative validity, the develop­ noise UIs, as well as identical levels of graphics quality. ment of questions involved consultation with four professional ar­ Both simulations (immersive and non-immersive) are formally clas­ chitects and three professional civil engineers, who provided general sified as exploratory simplified virtual reality. The term exploratory insights in the determination of the three-dimensional information of refers to when a user can perform visual search, exploring the virtual interest in collaborative design review, that is, the spatial relation­ environment at her/his will, defining her/his own path, stopping at ships in a project's 3D model that would be of interest to pro­ desired locations and focusing on certain objects. The term simplified fessionals involved, providing support to reasoning and feedback. refers to the degree of pictorial realism and vividness of a virtual envi­ For example, the structure team may be interested in the layout and ronment [77], i.e., its graphics quality [7,72]. In this study, both VR relative size of structural elements and spans proposed by the ar­ modes use identical low-fidelity simulations with limited rendering ef­ chitecture team (identifying and interpreting such visual information fects (static hard shadows and plain colors only). In summary, they only requires three-dimensional perception). These requisites were differ in interaction devices (keyboard and mouse vs. wireless control­ translated into 3DPQ's questions to better address what the design lers) and display type (laptop monitor vs. head-mounted display). IrisVR Prospect Plus was chosen as the application to run an Auto­ desk Revit BIM model in both niVR and IVR conditions. Hardware Table 1 ′′ comprised a high-performance 15 laptop, keyboard and mouse as 3DPQ's questions categories. interaction devices in the niVR setup. The IVR setup utilized the same Category Dimension Exploration Question laptop with an HMD (HTC Vive™) and two wireless controllers as # interaction devices. In summary, the conditions only differ in interaction Egocentric distance Depth Not allowed (fixed 1, 2, 3 devices (keyboard and mouse vs. wireless controllers) and display type estimation position) (laptop monitor vs. head-mounted display). Fig. 7 shows a set of Interobject distance Depth Not allowed (fixed 4, 5, 6 screenshots of the virtual environment in both VR modes. estimation position) Horizontal Allowed 7, 8, 9 Data collection instruments comprise: Vertical Allowed 10, 11, 12

Fig. 7. Screenshots of the virtual environment.

6 D. Paes et al. Automation in Construction 130 (2021) 103849

review team seeks over review meetings using 3D models. Questions subject designs than in between-subject designs given the same sample are multiple-choice, with objective alternative options comprising size [14,83]. intervals of estimates (e.g., up to 10 m, up to 12 m, and so on.) in lieu Therefore, a priori power analysis was conducted to provide the of Likert-scale “level-of-agreement” alternatives, in an attempt to necessary sample size for this study. As Gheisari [24] pointed out, HCI mitigate subjectivity in measurements of inherently subjective user studies usually verify whether the observed difference is real or random, experiences such as visual perception. Randomization software was the magnitude of the difference, and the meaningfulness of such used to sort the 3DPQ's questions to minimize eventual learning ef­ magnitude. Therefore, not only performing significance tests but also fects, originating three unique 3DPQs (all 3DPQs comprise the same defininga meaningful effect size is important to ensure the relevance of questions, but in different sequences). The 3DPQs are administered results. The definition of an adequate effect size – the estimated during walkthroughs in the virtual environments (questions and magnitude of the effect of a condition, i.e., the size of the difference alternative options are read out loud) and then in the visit to the between conditions – should be done in light of the research goals, as the physical environment. The 3DPQ can be provided to the reader upon meaningfulness of an effect magnitude can vary largely. While Cohen request. [15] provides standard values of effect magnitudes for studies in the social and behavioral sciences, effect sizes should be determined on a d. the Presence Questionnaire (PQ), used to collect the level of presence a case-by-case basis, as it is subject to the response variables, aims, in­ participant experienced during the 3D perception tasks in the virtual struments, and other specificaspects of a study. Frequently, large-sized environments (niVR and IVR). The PQ developed and used in this effects are chosen to ensure the practical significance of results [24]. research is predominantly based on the instrument developed by In this study, the effect size is positively related to the difference in Usoh et al. [86] named SUS (Slater-Usoh-Steed). However, it actually 3D perception between conditions and was estimated to ensure the comprises an adapted collection of questions from instruments of practical significanceof results. An initial percentage is computed into a Witmer and Singer [90], Usoh et al. [86], and Zikic [96], as well as statistics named odds ratio (OR; [82]) using the odds ratio formula, and one new question (question 7), and a final question on motion then the effect size is given by log(OR) (Table 2). Although small-sized sickness, in a total of eleven items. It adopts a 7-point Likert scale to effects may exist, these are not relevant to the ultimate purpose of this maintain consistency with the instruments of Witmer and Singer study – to verify the existence of effects of immersive visualization that [90] and Usoh et al. [86]. The decision of using the SUS as the main would benefit design review. Differences in 3D perception between reference was informed by previous studies [21] showing that conditions are only deemed beneficial/meaningful when considerably measures of presence have been more successful when metrics and high. Therefore, the effect size was computed from an expressive dif­ respective instruments address the three aspects of presence defined ference of 33% in accuracy scores between conditions (approximately 4 by Slater [76]: 1) the sense of being in the virtual environment, 2) the hits of difference out of 12 questions, since each question accounts for degree to which the virtual environment becomes the dominant re­ approximately 8% of difference), meaning an effect size of approxi­ ality, and 3) the extent to which the virtual environment is remem­ mately 0.69 (OR 4.88) as shown in Table 2 (scenario 4). In a similar bered as a ‘place’. Usoh et al. [86] and Slater's [76] approaches within-subject study, Ozcelik and Becerik-Gerber [64] adopted an effect appear to better address presence as a psychological state – or “state size of 0.7. of consciousness” [77] – resulting from virtual input rather than as a A priori power analysis utilizes the SIMR R package, which can direct function of system properties, such as argued by many scholars calculate power for Generalized Linear Mixed Models (GLMM) using regarding Witmer and Singer's [90] presence questionnaire. The PQ Monte Carlo simulations [28]. GLMM are suitable for the statistical can be provided to the reader upon request. analysis of count (number of occurrences), categorical (number of ob­ servations falling into separate categories), proportions, and continuous Following the development of the experimental design and Institu­ data, and are used in this study for hypothesis testing given the diverse tional Review Board (IRB) approval, an experiment simulation was characteristics of the data collected. Thus, power analysis should be conducted over one semester with 17 people to check for the adequacy conducted based on GLMM, which are the methods adopted for statis­ of the experimental design, data collection instruments, procedures, and tical data analysis. The SIMR runs a power analysis given a statistical equipment. This phase was particularly important to review and model and experimental design and calculates power curves to assess perform final adjustments to the experimental procedures and survey trade-offs between power and sample size. To perform the Monte Carlo questions. simulations, one must initially definean effect size of interest [28]. The For the actual experiment, participants were recruited in a non­ effect size estimated previously (0.69) is entered in the SIMR tool as the probabilistic manner (also referred to as nonrandom convenient sam­ magnitude of the difference expected to exist in the population. The pling). The population of interest consists of the body of industry SIMR then produces a chart showing the power of the test for several workforce (current and future workforce) traditionally involved in combinations of sample sizes and effect sizes (Fig. 8). Again, power is collaborative design review, including, but not limited to: architecture, calculated based on the categorical response variable of 3D perception. civil engineering, and building construction students, professional ar­ The calculated effect size yields a sample size of approximately 38 chitects, civil engineers, construction and facility managers, and trade participants on the 80% power curve provided by the SIMR R package contractors. This study investigates the effects of different treatments on power analysis (Fig. 8). There is no formal standard for the statistical this specific population – it is not concerned with the effects on the power, but 80% is a widely accepted value in the behavioral and social general population. sciences [28,83]. In sum, a sample of 38 participants has about 80% An a priori power analysis (prior to collecting data) should be con­ power to detect existing differences of 33% and up. Thus, the research ducted when a researcher is planning a study and wants to determine the power of a statistical test given a sample size if differences in the Table 2 response variable between conditions were similar to estimated values Effect size calculation. (effect size). Power is the probability that a particular statistical test will detect a difference when it exists, thus allowing for the correct rejection Scenario Condition Difference Odds Ratio (OR) Effect Size (log(OR)) of the null hypothesis [28]. Adopting a within-subject design contributes IVR niVR to increasing statistical power since each participant serves as her/his 1 0.58 0.5 0.08 (1 hit/12 q) 1.38 0.14 own equivalent hence excluding the variance between subjects due to 2 0.67 0.5 0.17 (2 hits/12 q) 2.03 0.31 individual differences in the comparison of effects of different condi­ 3 0.75 0.5 0.25 (3 hits/12 q) 3.00 0.48 tions. Studies show that power is generally much higher in within- 4 0.83 0.5 0.33 (4 hits/12 q) 4.88 0.69

7 D. Paes et al. Automation in Construction 130 (2021) 103849

5. Results

Preprocessing of data for statistical analysis involved cleaning up, coding, and organizing data for specific statistical software (PAST soft­ ware application – [29]). Inferential statistics are used to examine: a) whether differences in 3D perception and presence responses between experimental conditions, among questions, and between conditions per question are significant(H1 and H2), and b) the influenceof individual factors (confounding variables) and sequence/order of presentation of conditions on 3D perception and presence responses. The analysis utilizes several regression models to test the statistical hypotheses. P values are provided throughout data analysis to indicate the significanceof differences and relationships observed. According to past studies in the field, this research adopts a significance level (α) of 0.05, hence a confidence level of 95% (1 – α). Considering the entire sample, most participants are attending graduate school (94.7%) and 42.1% have between 26 and 33 years of age. Gender distribution in the sample is relatively and unintentionally balanced (female = 44.7%; male = 55.3%). In terms of their Bachelor's Fig. 8. Power curves for main effects. degree majors, 28.9% are Architects, 39.5% are Civil Engineers, and 5.3% are both. The participants have taken different professional roles team recruited 38 volunteers to take part in this study. during their industry experiences – oftentimes more than one occupa­ Participants were randomly assigned to all possible sequences of tion: 36.7% have worked as Architects, 39.4% as Civil Engineers, and conditions (randomization) so that each sequence is administered to the 18.3% as Construction Managers. Their experience in design review is same number of participants (counterbalancing). This is done to control relatively balanced across the first three ranges of experience: 21.1% for order effects and address systematic similarities across successive have up to 1 year, 26.3% have between 1 and 5 years, 23.7% have be­ conditions, which is the cause of practice/learning and fatigue effects tween 5 and 10 years, and 29% have over 10 years. All participants [52]. Kuliga et al. [50] highlight the possibility of a relationship be­ scored between 6 and 10 in the Spatial Ability Test (out of 10 questions). tween the effects of different virtual environments and the sequence in Over half the participants scored between 7 and 9 in a relatively which participants are exposed to these different simulations. In another balanced distribution, as follows: 15.8% scored 7 points, 21.1% scored 8 study, Ziemer et al. [95] examined how the order in which people points, 18.4% scored 9 points. Interestingly, 39.5% of the participants experience real and virtual environments influences their distance achieved a 10 score. judgments. In this study, there are two possible sequences of conditions: sequence 1 (niVR, IVR) and sequence 2 (IVR, niVR). An additional 5.1. H1 – Analysis of 3D perception strategy to mitigate learning effects consists of randomly sorting the order of survey questions across environments [68]. Besides conducting In regards to participants' 3D perception performance within the VR randomization, counterbalancing, and sorting the order of questions, modes, it was not expected that they would be accurate in perceiving allowing regular breaks during the experiment, as well as sufficient actual distances from the environment depicted. As previously dis­ acquaintance time and training so that participants can get used to cussed, experiments of this nature should not expect that participants navigation in the virtual environments can also help to reduce the are able to estimate, for instance, the actual ceiling height correctly. impact of learning and fatigue effects over consecutive conditions. Data Again, the initial goal is to compare a participant's 3D perception when analysis will also check for any order effects: any significantdifferences visiting the physical environment (reference mode) with her/his per­ in the dependent variables between groups that received treatments in ceptions in the virtual environments. Therefore, the answer to a question different order (group exposed to sequence 1 vs. group exposed to of the 3DPQ administered in each VR mode (3DPQ-niVR and 3DPQ-IVR) sequence 2). is compared to the response to the very same question administered in Data collection takes place in the experiment sessions. Each session the physical environment (3DPQ-PhE). The compatibility between these took approximately 70 min, in a total of 44 h of data collection over one responses indicates the ability of a virtual environment to reproduce 3D year. Fig. 9 shows participants performing perception tasks during perception obtained in the physical environment, hereafter referred to experiment sessions. as accuracy: the degree of resemblance to 3D perception in the real world. In other words, accuracy refers to the resulting deviation of a

Fig. 9. Participants performing perception tasks: in the IVR mode (left), in the niVR mode (center), and at the physical environment (right).

8 D. Paes et al. Automation in Construction 130 (2021) 103849 participant's 3D perception in a virtual environment with respect to her/ no difference in 3D perception between VR modes. his perception in the real world. The accuracy score is given by the ratio of hits over the total number of observations. A hit is definedas when a 5.1.1.2. Difference of global accuracy scores among questions. Differences participant selects the same alternative option to a given question of the of global accuracy scores among questions were also tested. This helps in 3DPQ in both reference mode and VR mode. Ziemer et al. [95] also the determination of the efficacy and validity of the 3DPQ question­ adopted the term accuracy score. Higuera-Trujillo et al. [35] developed naire. This analysis encompasses both conditions and focuses on eval­ a similar methodology, naming closeness what in this study is called uating how each question did in measuring the dependent variable. In accuracy. Their closeness scores were also obtained from the “stan­ this case, 3DPQ questions were included as predictor variables of ac­ dardization” of virtual-world responses over physical-world responses to curacy scores (simple linear regression). Fig. 11 shows the global ac­ simplify the comparisons among display formats. curacy scores per question. Pairwise comparisons (all possible pairs of Code numbers are assigned to hit and error categories: 1 represents a questions) show that the global accuracy scores are different within hit, and 0 represents an error/fail. Ozcelik and Becerik-Gerber [64] three pairs of questions. The global accuracy score of Q12 differs from conducted a similar study in which the first step was to compare par­ the global accuracy scores of Q3 (p = 0.017) and Q4 (p = 0.001), with ticipants' perceived temperature to the actual room temperature. They lower values found in Q12. The global accuracy score of Q4 also differs adopted 1 and 0 values to represent when a participant correctly guessed from the global accuracy score of Q8 (p = 0.015), with lower values in the actual temperature (within an error margin) or not, respectively. Q8. Questions 1, 2, 5, 6, 7, 9, 10 and 11 do not differ from any other Kimura et al. [46] also utilized 1 and 0 to represent a correct response question – neither from Q3 and Q4, which yielded the highest accuracy and an incorrect response, respectively. In this study, 1 and 0 values are scores, nor from Q12, which yielded the lowest score. The results indi­ derived from comparisons of participants' 3D perception between a cate consistency in the level of difficulty across 3DPQ questions. virtual environment and the real environment, not between real envi­ ronment and actual dimensions. Ultimately, these comparisons will 5.1.1.3. Difference of accuracy scores between conditions, per question and generate accuracy scores (per participant, per question, per condition), group of questions. The firstanalysis looks into the difference of accuracy which will support all subsequent analyses [68]. scores in the interaction between conditions and questions (Fig. 12). It Accuracy scores per VR modes are compared to determine which aims at examining the extent to which each question contributed to the condition better reproduced a participant's 3D perception in the real difference of global accuracy scores between conditions. Conditions, world. That is, which technology “did better” in reproducing a partici­ questions, and the interaction between them were included as predictor pant's 3D perception. The condition where participants achieve signifi­ variables of accuracy scores (multiple regression model with three cantly higher accuracy scores provides the most similar 3D perception to predictor variables). The accuracy scores differ between IVR and niVR that obtained in the real world. Thus, it could be deemed the technology conditions for questions Q3 (p = 0.002), Q5 (p = 0.027), and Q12 (p = that offers the most realistic 3D perception experience of built spaces. 0.002), with higher values found in the IVR condition. In all questions, the IVR condition shows higher accuracy scores, whether statistically 5.1.1. Significance test significant or not. The accuracy scores data set consists of categorical nominal values As expected, in each question separately the confidence interval is (hit/error values coded into one/zero values). This data set assumes a wider than the confidence interval of the global accuracy scores in the binomial probability distribution, which is adequate for treating cate­ VR modes, since the variance of data within each question in much gorical (nominal) responses. Three main relationships were tested: a) larger given the smaller data set per question (456 observations divided the difference of global accuracy scores between conditions, b) the dif­ by 12 questions, totalizing 38 data points per question). Therefore, the ference of global accuracy scores across questions, and c) the difference probability of detecting significant differences in each question sepa­ of accuracy scores between conditions, per question and group of rately was much lower. Nonetheless, in Q3, Q5, and Q12 the effect of questions. The letters above each group in the following charts represent conditions was so strong that even given a critically smaller data set and the comparisons. Groups that do not share the same letters are signifi­ larger variance, the difference between conditions was still significant. cantly different from each other (0.05 of significance). Such strong effects offset the other non-significant differences in other questions, resulting in a highly significantdifference of global accuracy 5.1.1.1. Difference of global accuracy scores between conditions. Condi­ scores between conditions. tions were included as predictor variables of accuracy scores and the The second analysis looks into the difference of accuracy scores be­ significance of the difference of global accuracy scores between condi­ tween conditions per group of questions. (Fig. 13). The questions were tions was tested (simple linear regression). Fig. 10 provides a chart divided into 4 groups according to their type (each row in Table 1), as showing the global accuracy scores in each VR mode, along with a table follows: Group 1) questions 1/2/3 of egocentric, depth, fixedestimation; with detailed information on the scores. The global accuracy scores Group 2) questions 4/5/6 of interobject, depth, fixedestimation; Group differ between IVR and niVR conditions (p < 0.001), with higher values 3) questions 7/8/9 of interobject, horizontal, exploration estimation; found in the IVR condition. Therefore, we refute the null hypothesis of Group 4) questions 10/11/12 of interobject, vertical, exploration esti­ mation. Conditions, group of questions, and the interaction between them were included as predictor variables of accuracy scores (multiple regression model with three predictor variables). Fig. 13 shows that

Fig. 10. Global accuracy scores in the VR modes. Fig. 11. Global accuracy scores per question.

9 D. Paes et al. Automation in Construction 130 (2021) 103849

5.2. H2 – Analysis of presence

The assessment of presence reported in the VR modes involves computing presence scores. A participant's presence score is the mean value across ten scores, keeping consistency with the instruments of Usoh et al. [86]. Figs. 14 and 15 provide histograms and descriptive statistics on the frequency of presence response values (1 to 7) in the niVR and IVR conditions, respectively.

5.2.1. Significance test The presence score per participant or per condition is a quantitative continuous variable representing the average of ten numbers ranging from 1 to 7 (a presence score can be any real number between 1 and 7). Therefore, the data set comprising presence scores can be approximated to a normal/Gaussian probability distribution. As in the analysis of 3D perception, three main relationships were tested: a) the difference of global presence scores between conditions, b) the difference of global presence scores across questions, and c) the difference of presence scores between conditions, per question. The letters above each group in the following charts represent the comparisons. Groups that do not share the same letters are significantly different from each other (0.05 of significance).

5.2.1.1. Difference of global presence scores between conditions. Condi­ Fig. 12. Accuracy scores between conditions, per question. tions were included as predictor variables of presence scores and the significance of the difference of global presence scores between condi­ tions was tested (simple linear regression). Fig. 16 provides a chart showing the global presence scores in each VR mode, along with a table with detailed information on the scores. The global presence scores differ between IVR and niVR conditions (p < 0.001), with higher values found in the IVR condition. Therefore, we refute the null hypothesis of no difference in presence between VR modes.

5.2.1.2. Difference of global presence scores among questions. Differences of global presence scores among questions were also tested. This helps in the determination of the efficacyand validity of the PQ questionnaire, as it consists of an adapted version based on previous instruments. This Fig. 13. Accuracy scores between conditions, per group of questions. analysis encompasses both conditions and focuses on evaluating how each question did in measuring the dependent variable. In this case, PQ questions were included as predictor variables of the presence scores participants did significantly better in IVR than in niVR in egocentric (simple linear regression). Fig. 17 shows the global presence scores per depth estimation at fixed position (Group 1, p = 0.002), in interobject question. Pairwise comparisons (all possible pairs of questions) show depth estimation at fixed position (Group 2, p = 0.016), as well as in that the global presence scores are different within several pairs of interobject vertical distance estimation at allowed navigation (Group 4, questions. The global presence scores of Q6, Q7, and Q9 are the lowest p = 0.032). among all questions and also significantly lower than the scores of: Q1 (Q1xQ6, Q1xQ7, Q1xQ9, p < 0.001), Q2 (Q2xQ6, Q2xQ7, Q2xQ9, p < 0.001), Q3 (Q3xQ6, Q3xQ7, Q3xQ9, p < 0.001), Q4 (Q4xQ6, Q4xQ7,

Fig. 14. Histogram of presence response values in the niVR condition.

10 D. Paes et al. Automation in Construction 130 (2021) 103849

Fig. 15. Histogram of presence response values in the IVR condition.

Fig. 16. Global presence scores in the VR modes.

Fig. 17. Global presence scores per question. Fig. 18. Presence scores between conditions, per question. Q4xQ9, p < 0.001), Q5 (Q5xQ6, Q5xQ7, p < 0.001; Q5xQ9, p = 0.003), and Q8 (Q8xQ6, Q8xQ7, Q8xQ9, p < 0.001). The global presence scores all questions). of Q6 and Q7 are also significantly lower than the score of Q10 As revealed in the previous analysis, Q6 and Q7 yielded significantly (Q6xQ10, p = 0.010; Q7xQ10, p = 0.006). Questions 1, 2, 3, 4, 5, 8 and lower global presence scores (both conditions considered) in relation to 10 do not differ among each other. Question 7 is the new question the other questions, meaning that participants reported significantly proposed in this study. lower presence with respect to the questions' statements. Nonetheless, The results indicate a slight inconsistency in the level of difficulty this analysis revealed that, at the same time, Q6 and Q7 also detected the across PQ questions. Question 7 – the one proposed by this study – was largest significant differences between conditions in favor of IVR (3.53 able to detect a global presence score of the same magnitude to those and 3.26, respectively). In other words, although participants reported detected by questions 6 and 9, which were found in the literature and lower presence levels with those questions, those were also the questions also differ from the remaining questions. Therefore, Q7 performed well that detected the largest differences in presence levels between condi­ in comparison to presence questions found in the literature. tions. It is worth noting that Q7 is the new question proposed by the research team. The results above reinforce the good performance of this 5.2.1.3. Difference of presence scores between conditions, per question. question. This analysis looks into the difference of presence scores in the inter­ action between conditions and questions (Fig. 18). It aims at examining 5.3. Analysis of individual factors & order effects on 3D perception and the extent to which each question contributed to the difference of global presence presence scores between conditions. Conditions, questions, and the interaction between them were included as predictor variables of pres­ This analysis verifiesthe influenceof individual factors (confounding ence scores (multiple regression model with three predictor variables). variables) and sequence/order of presentation of conditions on the 3D The presence scores differ between IVR and niVR conditions for all perception and presence variables. < questions, with higher values found in the IVR condition (p 0.001, for Confounding variables and order of presentation were included in

11 D. Paes et al. Automation in Construction 130 (2021) 103849 the models to check if they would distort the effects of conditions and Table 3 questions on the dependent variables. Two separate multiple regression Effects of individual factors and sequence on the dependent variables. models were adjusted for each response variable, i.e., one model pre­ Dependent variable dicting presence and a separate one for 3D perception. In summary, each 3D perception Presence model included conditions, questions, and sequence (controlled vari­ 2 2 ables) along with confounders, predicting a single dependent variable. It Confounding variables X P value X P value should be noted that confounding variables are included as predictor/ Bachelor's major 89,098.1 < 0.0 1.0 independent variables in the models. However, because these were 0.001 Experience in design review 3.8 0.052 1.6 0.204 actually not controlled/manipulated, any effects eventually found are Familiarity with the experiment 13.2 0.004 20.1 < restricted to this study's sample, not being generalizable to the study's environment 0.001 population. Spatial ability 0.0 0.959 2.5 0.116 Gender 0.0 0.935 9.5 0.002 5.3.1. Effects of individual factors It is only possible to include confounding variables in the model that Controlled variables are independent from each other. Therefore, the firststep of this analysis Conditions 20.0 < 0.001 153.5 < 0.001 was to identify which confounders (individual factors) are significantly Questions 50.4 < 0.001 177.5 < 0.001 associated with each other so that a single factor from a pair or cluster of Sequences 0.0 0.954 7.6 0.006 associated factors could be selected and included in the model as an independent variable. Naturally, the effects of factors that are associated conditions remained a significant factor in the explanation of 3D cannot be interpreted separately. For instance, if educational level is perception and presence. The effects of conditions on the dependent associated with age, and affects a dependent variable, it would not be variables were so strong that these remained significant even in the possible to attribute that effect to educational level only, as it could also presence of (or “controlled by”) those confounding variables. have been due to age. The individual factors examined are nine: age, gender, educational level, bachelor's major, current major, experience in 5.3.2. Order effects design review, experience with 3D virtual environments, familiarity In order to check if the sequence of presentation of conditions had with the experiment environment, and spatial ability. Because con­ any effect on the dependent variables, order/sequence was also included founding variables are categorical, evaluating association among them as a predictor/independent variable in the models. Order effects are any involves developing contingency tables of categorical data, also known significant differences in the dependent variables between groups that as tables of counts, and computing p values using the nonparametric Chi- received treatments in different order (group 1 – participants exposed to squared test of independence, which is adequate for treating categorical sequence 1 vs. group 2 – participants exposed to sequence 2). The order nominal data from independent samples [52]. The analysis of associa­ of presentation of conditions was manipulated as described previously. tion among individual factors revealed ten significant associations When included in the models as an independent variable along with among eight factors (gender – the ninth factor – was not associated with conditions, questions, and confounders, the effects of conditions and any other factor). Given this study's sampling process and sample questions on the dependent variables remain significant (Table 3). The characteristics, some significant associations were expected, for significant association between sequence and presence (p = 0.006) did instance, between age and educational level, age and experience in not alter the significanceof the effects of conditions and questions on the design review, and current major and familiarity with the experiment presence response. In other words, conditions remained a significant environment. factor in the explanation of 3D perception and presence. In sum, no The factors that had been selected to be tested for their impacts on order effect on 3D perception was detected, and the order effect on the dependent variables are five:gender (not associated with any other presence did not alter the effect of conditions on this variable. factor), bachelor's major, experience in design review, familiarity with the experiment environment, and spatial ability. The decision on which factors to select within a pair or cluster of associated factors is made 6. Discussions and contributions based on their relevance in light of the general research goals. The selected confounders were deemed as better predictors of the response The next sections present a detailed discussion of results and the variables for different reasons. For instance, bachelor's major, which is main contributions of this study in light of previous research. interpreted as one's main professional role or original fieldof practice, is selected over educational level because the study's population of interest 6.1. Discussions consists of professionals whose roles are primarily defined by their bachelor's majors (architect, civil engineer, etc.), not their educational Regarding H1, the comparison of accuracy scores between VR modes levels. Another example is the decision between experience with 3D indicates that IVR technology can better reproduce a user's 3D percep­ virtual environments and spatial ability. While both could affect 3D tion in the real world than non-immersive VR. Participants achieved a perception in virtual environments, the former is a rather subjective, significantlyhigher accuracy score in the IVR condition, meaning that it self-reported response, whereas spatial ability was measured through a provided the most similar 3D perception to that obtained in the real more objective method and hence is seeing as more reliable. world (i.e., the closest-to-reality experience). In order words, IVR can be The inclusion of individual factors and order of presentation in the deemed the technology that offers the most realistic 3D perception models revealed the significance of their effects on the dependent var­ experience of built spaces. Using the immersive technology participants iables. Significance values are provided in Table 3. had a better 3D perception of the architectural representation compared The significant individual factors for 3D perception were bachelor's to their perception using the conventional workstation. The IVR system major (p < 0.001) and familiarity with the experiment environment (p allowed users to perceive three-dimensional features more accurately. = 0.004). For presence, significant confounders were familiarity with IVR appears to be the most appropriate system for tasks that benefit the experiment environment (p < 0.001) and gender (p = 0.002). The from accurate representation and communication of three-dimensional significanteffects of confounding variables did not alter the significance spaces. The perception of three-dimensionality was operationalized in of the effects of conditions and questions on the dependent variables. this study as one's distance estimates. The greater accuracy in distance That is, even under significant influence of individual factors, the dif­ estimation using the immersive platform implies a better understanding ference between conditions remains significant. In other words, of the three-dimensional configurationof the space depicted. Whether a

12 D. Paes et al. Automation in Construction 130 (2021) 103849 user is able to better estimate distances – when one's virtual estimates The analysis of individual factors on 3D perception and presence are closer to real-world estimates – it means a better understanding of indicated that the effects of conditions and questions on the dependent the three-dimensional configuration of the virtual model. An enhanced variables remained significant even in the presence of (or “controlled 3D perception can be interpreted as a better understanding of the three- by”) confounding variables (individual traits). In other words, condi­ dimensional relationships of the architectural representation, meaning tions remained a significant factor in the explanation of 3D perception that within the immersive environment, geometric information “makes and presence. Also, order effects were either not detected or did not more sense” and can be better assimilated by the observer. In summary, impact the effects of conditions and questions on both response vari­ the representation and communication of three-dimensional informa­ ables, confirming that the randomization and counterbalancing pro­ tion are leveraged with the support of the immersive system. Ultimately, cedures were effective. While Kovac [49] suggests that spatial ability is a better understanding of the virtual model is expected to benefitdesign positively correlated with users' performance in spatial tasks, both review and increase productivity levels. spatial ability and experience in design review did not show any sig­ Participants did significantly better in IVR than in niVR in three of nificant effects on the dependent variables, contrary to the researcher's four types of distance judgments – egocentric and interobject depth expectations. However, this finding is consistent with the expectations estimation at fixed position, as well as in interobject vertical distance of Witmer and Singer [90] and Calderon-Hernandez et al. [10]. estimation at allowed navigation. This result suggests that, in general, Regardless of the strategies adopted to mitigate the problem of sys­ people estimate distances more accurately in the immersive environ­ tematic similarities between conditions, participants are not likely to ment regardless of their vantage point (allowed exploration or fixed learn over successive exposure to experimental conditions because these position) and distances being estimated (egocentric or interobject). are not exactly similar, and they are made aware of that. In this study, As indicated by the analyses of the difference of accuracy scores the extent to which early trials induce a response bias that influence among questions, and between conditions per question, familiar archi­ performance on later trials is reduced by asking participants to “rethink tectural elements such as doors and staircase steps may also help in 3D their answers” since stimulus varies across conditions [83]. Participants perception, regardless of the VR mode. Participants strongly relied on are not told the correct answer after being exposed to the firstcondition; their previous knowledge about such elements in the environment, that is, they do not know if their guesses in early trials are right or wrong which ended acting as major depth cues in both conditions. so that there is no useful information that could “prime” or influence Although the technological and representational factors in charge of their decisions in the following condition. In other words, participants promoting better 3D perception in the immersive system were not could not consciously use their guesses in the previous condition as a investigated separately in this study, stereopsis – a visual cue specificto reference to try to guess more accurately in the following condition, and the immersive condition – is most likely among them. Other system they are made aware of that. In this study, being consecutively exposed properties such as field of view and interaction devices may also have to stimulus does not make a participant more likely to be more precise in affected 3D perception. However, it is impossible to isolate stereopsis the next condition simply because stimuli are fundamentally not similar from those due to the nature of VR systems utilized in this study. While (different media cause different stimulus). Participants had the chance Kalisperis et al. [41] and Zikic [96] focused on measuring the effects of and were oriented to reevaluate their responses, which was precisely the technological characteristics of immersive systems such as display fea­ goal of the experiment. Even though they may recall their answers in tures (e.g., stereoscopy, screen size, field of view) and representational early trials, they know that stimulus varies across conditions, so their aspects (e.g., level of realism, level of detail) on perception and pres­ previous responses should not be assumed correct. ence, this study provides the global efficiency of VR systems given all their fixed factors. Regardless, results suggest that the immersive envi­ 6.2. Contributions ronment provided critical depth cues to enable a more realistic depth perception, unlike depth cues available in the niVR condition. Since The characteristics of previous studies (e.g., aims, context, approach, featural cues (representational aspects) were kept the same across con­ target population, methodology, experimental design, data analysis ditions, stereopsis is likely one of the cues responsible for that difference. methods) from various domains (e.g., built environment, cognitive While occlusion and perspective are important depth cues [17], results psychology, HCI) addressing presence or perception in virtual environ­ suggest that stereopsis may also play a crucial role in depth judgments in ments vary largely (see Sections 1, 3 and 4.1). Due to the inherent virtual environments, confirmingthe expectations of England et al. [20]. limitations of their methodologies or experimental designs, they have In a scenario where stereopsis is provided in addition to occlusion and been unable to provide a complete answer to the long-lasting question of perspective (as in the immersive condition), participants report more whether and the extent to which immersive systems can deliver more accurate depth perception. effective representations from the user standpoint and greater levels of Regarding H2, results suggest that IVR technology offers greater presence than non-immersive media in the particular activity of design levels of presence, meaning that it provided more immersive experiences review. Therefore, the assumption that immersive environments are and confirmingthe findingsof Witmer and Singer [90], Kalisperis et al. inherently more effective than non-immersive media in conveying [41], Zikic [96], and Castronovo et al. [12]. IVR appears to be the most geometric information and promoting the sense of presence (in the appropriate system when a VR-supported task benefitsfrom immersion design review activity, specifically) was highly contentious, given the and involvement, such as design review. Greater levels of presence are lack of supporting robust evidence. For instance, while previous studies expected to enhance a user's ability to perform visual search and un­ may have included comparisons, these were often qualitative or derstand the displayed information while interacting with a design exploratory, which normally do not allow for the generalization of re­ representation hence facilitating the identification of design issues and sults across the study population. They may have been experimental but ultimately benefiting the design review process. underpowered (insufficient statistical power). Also, they may not have Significantdifferences in some questions were expected, but not for considered the effects of order and individual characteristics on the all questions as revealed by this analysis. In every question, presence in response variables. In general, they all present some critical limitation IVR was significantly greater than in niVR. The analysis shows that (s). These issues are discussed in detail in Sections 2 and 4.1. large-sized significantdifferences in presence scores between conditions Thus, this work's key contributions stir from its methodology. The could have been detected with a single question (any question), which innovative combination of five methodological characteristics, namely, reinforces the fact that if power analysis had been done based on the approach, context, comparison, controlled experiment, and inferential presence score variable, the sample size would have been much smaller, statistical analysis, was able to generate robust and generalizable evi­ thus compromising the power to detect differences in the other response dence about the existence and extent of improvements to 3D perception variable of 3D perception. and presence levels of a specific user population, eliciting the

13 D. Paes et al. Automation in Construction 130 (2021) 103849 effectiveness of immersive representations in design review. Previous Inferential statistical analysis: This study adopts quantitative data studies would often overlook one or more of the aspects above covered analysis methods, including inferential statistics, which allows for the by this research. A summary of the strategies adopted in this study to generalization of results across the user population and technology address past studies' limitations is provided below. application context. The literature review pointed out that the vast Approach (user-centered): This study exercises a user-centered majority of past studies have either not conducted or provided a clear approach as opposed to a technology-centered one. The effectiveness description of an a priori power analysis, which is vital to defining the of VR systems is given by the degree to which they successfully promote adequate sample size to provide a statistical test with sufficientpower to accurate 3D perception and high levels of presence. Ultimately, the detect existing differences. If enrolling too few participants, a study may advantages of using IVR to represent and display architectural artifacts not have enough statistical power to detect differences when they exist are only valid if the technology improves the users' 3D perception of the (type II error). This may jeopardize the validity of statistical results and virtual environment and presence levels during the review task. This an investigator's ability to make inferences. The effect size of interest – condition must be satisfied to establish the benefits of IVR technology the magnitude of the difference between conditions – should also be for design review. carefully determined during the power analysis. Effect sizes should be Context (design review): An experimental study's findings are only determined on a case-by-case basis, as they are subject to the response applicable to its target population and experimental setup. Therefore, variables, aims, instruments, experimental design, and even sampling research in the architecture and built environment domains should not factors of a study. Only one study found in the literature provides an fully rely on studies from other fields that have targeted the general effect size with no clear description of how it has been calculated [64], population or utilized different technological setups from the ones which poses a threat to the practical significanceof differences that past normally used in the built environment. This study's controlled experi­ studies may have eventually detected. ment has been designed to reproduce the technology application envi­ Furthermore, as opposed to previous studies, this study's analyses ronment to ensure that findings had practical significance by targeting cover not only the differences of 3D perception and presence scores the population of professionals traditionally involved in collaborative between conditions but also across questions (to show how each ques­ design review, utilizing a relevant technological setup (BIM model dis­ tion did in measuring the dependent variables) and between conditions played through laptop monitor and off-the-shelf HMD), and simulating per question (to reveal the extent to which each question contributed to tasks normally performed using the technology (design review). the difference of perception and presence scores between conditions). Comparison (immersive vs. non-immersive): Comparisons and bench­ These analyses are vital to check the validity of the adapted versions of marking are critical to check the validity and justify the adoption of a the PQ and 3DPQ scales utilized in this study. Although developing new given technology over others. Generally, new tools and methods can instruments was not among the main goals of this research, it may still only be deemed more effective in comparison to their peers or previous contribute to this matter since the assessment of the adapted versions ones. That is why, as opposed to self-evaluations and inspection-based suggests internal consistency and validity of instruments. assessments, comparative investigations facilitate the observation of improvements – as these can only be established against something else 7. Conclusions – providing a means for determining the level of effectiveness of new solutions. Thus, this study compares an immersive VR system to a non- IVR effectiveness over traditional VR in design review has been immersive one. repeatedly reported in the past. However, the vast majority of such Controlled experiment / experimental design: Exploratory and qualita­ studies demonstrating IVR implications on productivity and task per­ tive studies were crucial to lay down the foundations of research in the formance often delivered a rather anecdotal report of users' experiences field. However, experimental research appears to be the only approach with IVR systems, disregarding the eventual cognitive improvements that enables researchers to judge beliefs and assumptions with system­ provided by such technology. A fundamental step for IVR to be atically measured confidence and reliability [52]. considered an effective technology for collaborative design review is to When the purpose of a virtual environment is to reproduce human understand whether and to what extent it enhances users' cognitive cognition in the real world, its effectiveness must be established by abilities, that is, their performance in obtaining and assimilating the comparing user performance in the virtual setup against performance in spatial relationships depicted. Cognitive processes are in charge of the real world. Thus, in studies that compare ‘spatial’ perception per­ human performance; therefore, VR effectiveness assessments must formance between different virtual environments, estimates in the consider the cognitive processes that underlie user performance in vir­ physical world should be used to standardize virtual estimates first. tual environments. In conformity with the main references in HCI and Therefore, as an improvement from previous comparative methods and cognitive psychology, the effectiveness analyses of virtual environments based on studies from the cognitive sciences, a participant's 3D provided by this study have been conducted on measures of user per­ perception in the real world was collected and directly compared to 3D formance. User-centered research on IVR systems is relevant to both perception in the virtual environments. The experiment was designed to industry and academia to the extent that it could ultimately lead to novel investigate whether a participant's 3D perception in the VR modes de­ collaborative IVR environments and methods tailored to specific users, viates from perception in the real environment, regardless of what es­ purposes and application contexts. timates were reported in the real world. Thus, the method is at the same This study expands the characterization of the benefits repeatedly time: a) not concerned whether real-world estimates are accurate in reported in the past. To “assess the extent to which IVR offers better relation to the actual dimensions, and b) taking into account these es­ support to design review compared to non-immersive VR platforms” timates to standardize VR estimates. In short, the assessment of 3D means to examine the phenomenon that is causing people to better perception in the VR modes is done by comparing it to 3D perception in understand a 3D model using IVR technology. This research provides a the real world. This comparison provides a VR system's effectiveness in deeper analysis of what is actually happening when a user reports a terms of its ability to simulate a given space from the user standpoint. better understanding of a project's 3D model through immersive visu­ Another critical limitation addressed is the omission concerning the alization. As per Wann and Mon-Williams [88], the effectiveness of effects of order (sequence of exposure to conditions) and individual virtual environments is dictated by perceptual criteria. Thus, this study differences (including spatial ability), which must be controlled through focused on the characterization of the perceptual experiences that un­ the experimental design and/or statistical analysis. In this study, the derlie the increased IVR effectiveness over traditional VR. effects of order and individual factors were controlled through a within- The factors that lead users to report increased effectiveness are not subject design and checked for any effects on the response variables directly related to the VR system per se, but to the cognitive processes through statistical analysis. that have been enhanced by the technology (Fig. 2). Evidence shows that

14 D. Paes et al. Automation in Construction 130 (2021) 103849 when people report that IVR is more efficientin the design review, this is technology would also provide a better 3D perception of virtual envi­ related to a better understanding of the spatial relationships of a 3D ronments with different spatial properties/depth cues from those pre­ model. Based on the literature, this study hypothesized that reported sent in the lobby model utilized in this study. benefits are associated with an enhanced 3D perception of the design Whenever the goal is to compare the effects of different visualization representation and greater involvement in the review activity. Hy­ methods on visual perception, the viewed space or object should be the potheses tests were conducted and confirmed that the IVR system pro­ same across them. Comparing the perception of 3D models of different vides enhanced 3D perception and greater levels of presence. This study spaces could be challenging, as spatial properties will likely vary if did not cover technological factors that may contribute to those benefits. spaces are different. In this case, one could not attribute eventual dif­ However, it provides evidence that 3D perception and presence in the ferences in perception to the visualization tool, as these could be due to review of 3D models are improved using IVR technology compared to the presence or absence of critical depth cues, as explained above. non-immersive VR systems. Furthermore, it would be hard – if not impossible – to isolate and In this context, the practical significance of the differences between manipulate depth cues to observe their individual effects on perception technologies lies in the relevance of such differences in light of the (or any other response variable). design practice, and it is given by the high probability of finding large If the goal is to determine the effects of depth cues on perception, one improvements in the study's population. Although small-sized effects may manipulate the presence/absence of each cue to create different 3D may exist, these were deemed not relevant to the ultimate purpose of models of the same space (i.e., two slightly different versions of the same this study – to verify the existence of effects of immersive visualization model) and then compare the perception of those models. The experi­ that would significantly benefit design review. Thus, a priori power mental design of such studies could vary largely depending on the analysis was conducted based on a meaningful and considerably high research questions, the number of experimental conditions/treatments difference in 3D perception between technologies, ensuring an adequate (model versions), and response variables to be measured. Providing a sample size for a high probability (80% power) to detect existing dif­ detailed reference framework for future studies is not within the scope of ferences of 33% and up. One can expect to find such improvements in this paper, and the reader may refer to Lazar et al. [52] for instructions the population as well. In sum, results provide practical significance to on how to design experiments in HCI research. Experiment-based studies the extent that these indicate a high probability to find large-sized im­ could reach high levels of complexity (see [41]) to the point of becoming provements in the population, which is relevant to the practice. The impractical given the number of conditions and sample size. Therefore, ability of IVR technology in providing users with significantlybetter 3D special attention should be paid to the design of experiments to ensure perception is expected to improve the understanding of 3D models and, these provide the relevant data to answer the research questions. consequently, collaborative design review. Another recommendation is to restrict experiments to only a few It should be noted that a better understanding of the three- experimental conditions and collecting a small number of response dimensionality of a virtual model and greater levels of presence in the variables. An experiment should involve at least two conditions (a single simulation do not necessarily imply a better understanding of a de­ independent variable could assume different values in each condition, signer's intentions nor smarter design solutions. Nonetheless, it would such as “texture vs. no texture”). In this example, the experiment aims to represent a critical step towards it since architectural design is mainly determine whether changes in texture induce changes in the response concerned with solving spatial problems. It is reasonable to expect that variable (e.g., perception). once the three-dimensionality of a virtual model is better understood As discussed by Berg and Vance [6] and Paes and Irizarry [67], the and greater levels of presence are reached, more suitable solutions to level of realism of a simulation is strictly a function of the questions those spatial problems are likely to arise. However, the cause-effect being explored and goals with that simulation. A researcher wanting to relationship among enhanced 3D perception, levels of presence, and investigate interceptive timing behavior might wish to violate Newto­ quality of design solutions could not be demonstrated through this study nian mechanics so that objects can move in unexpected trajectories [9]. and may be addressed in future research. In that case, an unrealistic simulation (with respect to the rules of Furthermore, while different equipment may prompt different re­ Newtonian mechanics) is necessary. Nonetheless, the power to resemble sponses from the ones observed in this study, user-centered studies in the the world as people see and interact with is precisely what makes IVR fieldshould not focus on the systems, devices, or equipment, but on the technology a promising tool to simulate, predict, and investigate critical user experience with the technology. Because technology is rapidly implications of architectural design solutions according to the reality evolving and equipment is quickly becoming obsolete, if a validation that people are able and used to experience. study of different platforms is applicable, this should be done in light of Given that the level of realism of a simulation is application- user experiences. Nonetheless, the particular combination of hardware dependent, future research may also look into the visual and interac­ and software utilized in this study restricts the research findings to its tive features required in immersive simulations for optimal cognitive particular technological setup. Also, the results obtained in this research performance and productivity, that is, explore and identify what visual may be restricted to the particularities of the modeled environment, so cues (shading, texture, occlusion, stereopsis, parallax, motion perspec­ that when altering spatial properties or analyzing a different space re­ tive, etc.) and equipment properties (display resolution, latency, field- sults could change. As future work, it would be interesting to replicate of-view, interfaces, etc.) would deliver optimal responses (not neces­ this study using other 3D models. sarily presence and perception) and benefita given application the most Spatial properties are physical characteristics of an environment (e.g., architectural design, safety training, etc.). One may find out an directly associated with the depth cues provided (see Section 3.1). For equation that allows the tradeoff among visual cues, technological and example, a reflectiveceramic floormay afford depth cues that are absent human factors towards enhancing user performance in a virtual envi­ in a concrete floor. These spatial properties and associated depth cues ronment. The path towards such an equation starts by understanding the may facilitate or hinder depth perception. For instance, whether a given depictive information needs in each application context, which is space (or its virtual counterpart) is not well lit or does not have a grid or another entirely independent topic for future research. geometric floortexture will likely impact one's ability to perceive depth Regardless of the contributions of this work, research in the built accurately. Therefore, “when altering spatial properties or analyzing a environment fieldhas yet to demonstrate the benefitsof IVR systems to different space” from the one examined in this study, the absolute values other aspects of the design process and other construction tasks. In this of 3D perception accuracy scores may change, however, not to the point study, the approach was to test whether and to what extent IVR systems of contradicting the overall better perception using the IVR system could deliver more effective representations than traditional systems in (given that scores are standardized over real-world perception re­ terms of users' 3D perception and presence responses. Future studies sponses). A recommendation for future research is to test whether IVR may choose different approaches such as investigating the combination

15 D. Paes et al. Automation in Construction 130 (2021) 103849 of IVR and other representation formats, the adequacy of design solu­ [18] P.S. Dunston, L.L. Arns, J.D. Mcglothlin, G.C. Lasker, A.G. Kushner, An Immersive tions developed in immersive environments, or communication effi­ Virtual Reality Mock-Up for Design Review of Hospital Patient Rooms. Collaborative Design in Virtual Environments. Intelligent Systems, Control and ciency in IVR-supported review meetings. Other studies could involve Automation: Science and Engineering 48, Springer, Dordrecht, 2011, pp. 167–176, user-centered validation of IVR systems to study workers' behavior in https://doi.org/10.1007/978-94-007-0605-7_15. hazardous construction environments, construction accident response, [19] C. Eastman, P. Teicholz, R. Sacks, K. Liston, BIM Handbook: A Guide to Building Information Modeling for Owners, Managers, Designers, Engineers, and disaster evacuation, heavy equipment operation, or occupants' in­ Contractors, John Wiley & Sons, Hoboken, 2008. ISBN: 978-0-470-18528-5. teractions with building systems – particularly in specialized facilities. [20] R.D. England, G.K. Edgar, K.T. Carr, Human Factors in Virtual Worlds: (2) Enabling Interaction and Performance. Proceeding of the IEE Colloquium on Using Virtual Worlds, London, 1992, pp. 4/1–4/5. Available at: https://ieeexplore.ieee. org/document/168238. Declaration of Competing Interest [21] D. Faas, Q. Bao, D.D. Frey, M.C. Yang, The influenceof immersion and presence in early stage engineering designing and building, Artif. Intell. Eng. Des. Anal. Manuf. – The authors declare that they have no known competing financial 28 (2) (2014) 139 151, https://doi.org/10.1017/S0890060414000055. [22] T. Fernando, K.C. Wu, M. Bassanino, Designing a novel virtual collaborative interests or personal relationships that could have appeared to influence environment to support collaboration in design review meetings, J. Inf. Technol. the work reported in this paper. Constr. (ITcon) 18 (2013) 372–396 (ISSN: 1874-4753), https://www.itcon. org/paper/2013/19. [23] W. Florio, Analise´ do processo de projeto sob a teoria cognitiva: sete dificuldades Acknowledgments no atelier (An analysis of the design process under the cognition theory: seven difficulties in the atelier), Arquiteturarevista 7 (2) (2011) 161–171, https://doi.org/10.4013/ arq.2011.72.06. This work was partially supported by the Brazilian National Council [24] M. Gheisari, An Ambient Intelligent Environment for Accessing Building of Technological and Scientific Development (CNPq) [grant number Information in Facility Management Operations; a Healthcare Facility Scenario, 203359/2014-3]. Doctoral Dissertation, Georgia Institute of Technology, Atlanta, United States, 2013. Available at: http://hdl.handle.net/1853/52967. [25] J.J. Gibson, The Ecological Approach to Visual Perception, Houghton Mifflin, References Boston, 1979. ISBN: 1848725779. [26] R. Gifford, Environmental Psychology: Principles and Practice, Optimal Books, Colville, 2002. ISBN: 0968854303. [1] M. Adi, D. Roberts, Using virtual environments to test the effects of lifelike [27] A.A. Gooch, P. Willemsen, Evaluating Space Perception in NPR Immersive architecture on people, Technol. Incl. Well-Being 536 (2014) 261–285, https://doi. Environments. Proceedings of the 2nd International Symposium on Non- org/10.1007/978-3-642-45432-5_13. Photorealistic Animation and Rendering, 2002, pp. 105–110, https://doi.org/ [2] A. Al-Hammad, S. Assaf, M. Al-Shihah, The effect of faulty design on building 10.1145/508530.508549. maintenance, J. Qual. Maint. Eng. 3 (1) (1997) 29–39, https://doi.org/10.1108/ [28] P. Green, C.J. MacLeod, SIMR: an R package for power analysis of generalized 13552519710161526. linear mixed models by simulation, Methods Ecol. Evol. 7 (4) (2016) 493–498, [3] M. Bassanino, K. Wu, J. Yao, F. Khosrowshahi, T. Fernando, J. Skjaerbaek, The https://doi.org/10.1111/2041-210X.12504. Impact of Immersive Virtual Reality on Visualization for a Design Review in [29] Ø. Hammer, D.A.T. Harper, P.D. Ryan, PAST: paleontological statistics software Construction. Proceedings of the 14th International Conference on Information package for education and data analysis, Palaeontol. Electron. 4 (1) (2001) 1–9. Visualization, 2010, pp. 585–589, https://doi.org/10.1109/IV.2010.85. ISSN: 1094-8074. Available at: http://palaeo-electronica.org/2001_1/past/issue1 [4] B. Becerik-Gerber, F. Jazizadeh, N. Li, G. Calis, Application areas and data _01.htm. requirements for BIM-enabled facilities management, J. Constr. Eng. Manag. 138 [30] D. Henry, T. Furness, Spatial Perception in Virtual Environments: Evaluating an (3) (2012) 431–442, https://doi.org/10.1061/(ASCE)CO.1943-7862.0000433. Architectural Application. Proceedings of the IEEE Virtual Reality Annual [5] L.P. Berg, J.M. Vance, An industry case study: investigating early design decision International Symposium (VRAIS '93), Seattle, 1993, pp. 33–40, https://doi.org/ making in virtual reality, J. Comput. Inf. Sci. Eng. 17 (1) (2016) 011001–011007, 10.1109/VRAIS.1993.380801. https://doi.org/10.1115/1.4034267. [31] A. Heydarian, J.P. Carneiro, D. Gerber, B. Becerik-Gerber, T. Hayes, W. Wood, [6] L.P. Berg, J.M. Vance, Industry use of virtual reality in product design and Immersive Virtual Environments: Experiments on Impacting Design and Human manufacturing: a survey, Virtual Reality 21 (1) (2016) 1–17, https://doi.org/ Building Interaction. Proceedings of the 19th International Conference on 10.1007/s10055-016-0293-9. Computer-Aided Building Design Research in Asia (CAADRIA), Kyoto, 2014, [7] D. Bertol, Designing Digital Space: An Architect’s Guide to Virtual Reality, John pp. 729–738. ISBN: 978-988-19026-5-8. Wiley & Sons, New York, 1997. ISBN: 0-471-14662-5. [32] A. Heydarian, J.P. Carneiro, D. Gerber, B. Becerik-Gerber, Immersive virtual [8] C. Boton, Supporting constructability analysis meetings with immersive virtual environments, understanding the impact of design features and occupant choice reality-based collaborative BIM 4D simulation, Autom. Constr. 96 (2018) 1–15, upon lighting for building performance, Build. Environ. 89 (2015) 217–228, https://doi.org/10.1016/j.autcon.2018.08.020. https://doi.org/10.1016/j.buildenv.2015.02.038. [9] J. Brookes, M. Warburton, M. Alghadier, M. Mon-Williams, F. Mushtaq, Studying [33] A. Heydarian, J.P. Carneiro, D. Gerber, B. Becerik-Gerber, T. Hayes, W. Wood, human behavior with virtual reality: the Unity experiment framework, Behav. Res. Immersive virtual environments versus physical built environments: a Methods (2019) 1–9, https://doi.org/10.3758/s13428-019-01242-0. benchmarking study for building design and user-built environment explorations, [10] C. Calderon-Hernandez, D. Paes, J. Irizarry, X. Brioso, Comparing Virtual Reality Autom. Constr. 54 (2015) 116–126, https://doi.org/10.1016/j. and 2-Dimensional Drawings for the Visualization of a Construction Project. ASCE autcon.2015.03.020. International Conference on Computing in Civil Engineering (i3CE 2019), Atlanta, [34] A. Heydarian, J.P. Carneiro, E. Pantazis, D. Gerber, B. Becerik-Gerber, Default 2019, pp. 25–31, https://doi.org/10.1061/9780784482421.003. Conditions: A Reason for Design to Integrate Human Factors. Proceedings of the [11] F. Castronovo, D. Nikolic, Y. Liu, J.I. Messner, An Evaluation of Immersive Virtual First International Symposium on Sustainable Human-Building Ecosystems, Reality Systems for Design Reviews. Proceedings of the 13th International Pittsburgh, 2015, pp. 54–62, https://doi.org/10.1061/9780784479681.006. Conference on Construction Applications of Virtual Reality (CONVR 2013), [35] J.L. Higuera-Trujillo, J. Lopez-Tarruella Maldonado, C.L. Millan, Psychological and London, UK, 2013, pp. 22–29. ISBN: 978-0-9927161-0-3. physiological human responses to simulated and real environments: a comparison [12] F. Castronovo, D. Oprean, T. Liu, J. Messner, Application of Immersive Virtual ◦ between photographs, 360 panoramas, and virtual reality, Appl. Ergon. 65 (2017) Reality Systems in an Interdisciplinary Design Studio Course. Proceedings of the 398–409, https://doi.org/10.1016/j.apergo.2017.05.006. Joint Conference on Computing in Construction (JC3), Heraklion, Greece, 2017, [36] G.S. Hubona, G.W. Shirah, D.G. Fout, The effects of motion and stereopsis on three- pp. 337–345, https://doi.org/10.24928/JC3-2017/0006. dimensional visualization, Int. J. Hum. Comput. Studies 47 (5) (1997) 609–627, [13] S.K. Chandrasegaran, K. Ramani, R.D. Sriram, I. Horvath,´ A. Bernard, R.F. Harik, https://doi.org/10.1006/ijhc.1997.0154. W. Gao, The evolution, challenges, and future of knowledge representation in [37] G.S. Hubona, P.N. Wheeler, G.W. Shirah, M. Brandt, The relative contributions of product design systems, Comput. Aided Des. 45 (2) (2013) 204–228, https://doi. stereo, lighting, and background scenes in promoting 3D depth visualization, ACM org/10.1016/j.cad.2012.08.006. Trans. Comput. Hum. Int. 6 (3) (1999) 214–242, https://doi.org/10.1145/ [14] G. Charness, U. Gneezyb, M. Kuhn, Experimental methods: between-subject and 329693.329695. within-subject design, J. Econ. Behav. Organ. 81 (1) (2012) 1–8, https://doi.org/ [38] R. Ibrahim, F.P. Rahimian, Comparison of CAD and manual sketching tools for 10.1016/j.jebo.2011.08.009. teaching building design, Autom. Constr. 19 (8) (2010) 978–987, https://doi.org/ [15] J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd edition, 10.1016/j.autcon.2010.09.003. Lawrence Erlbaum Associates, Hillsdale, 1988. ISBN: 0-8058-0283-5. [39] V. Interrante, B. Ries, L. Anderson, Distance Perception in Immersive Virtual [16] C. Cruz-Neira, D.J. Sandin, T.A. DeFanti, Surround-Screen Projection-Based Virtual Environments, Revisited. Proceedings of the IEEE Virtual Reality Conference (VR Reality: The Design and Implementation of the CAVE. Proceedings of the 20th 2006), Alexandria, 2006, pp. 3–10, https://doi.org/10.1109/VR.2006.52. Annual Conference on Computer Graphics and Interactive Techniques, Anaheim, [40] V. Interrante, B. Ries, J. Lindquist, M. Kaeding, L. Anderson, Elucidating factors 1993, pp. 135–142, https://doi.org/10.1145/166117.166134. that can facilitate veridical spatial perception in immersive virtual environments, [17] J.E. Cutting, P.M. Vishton, Perceiving Layout and Knowing Distances: The Presence 17 (2) (2008) 176–198, https://doi.org/10.1162/pres.17.2.176. Integration, Relative Potency, and Contextual Use of Different Information About [41] L. Kalisperis, K. Muramoto, B. Balakrishnan, D. Nikolic, N. Zikic, Evaluating Depth. Perception of Space and Motion, 2nd edition, Academic Press, 1995, Relative Impact of Virtual Reality System Variables on Building Design pp. 69–117. ISBN: 0-1224-0530-7.

16 D. Paes et al. Automation in Construction 130 (2021) 103849

Comprehension and Presence: A Variable-Centered Approach Using Fractional [65] D. Paes, J. Irizarry, Virtual Reality Technology Applied in the Building Design Factorial Experiment. Proceedings of the 24th Education and Research in Process: Considerations on Human Factors and Cognitive Processes. Advances in Computer Aided Building Design in Europe (eCAADe), Volos, 2006, pp. 66–73. Ergonomics in Design, 2016, pp. 3–15, https://doi.org/10.1007/978-3-319-41983- ISBN: 0-9541183-5-9. 1_1. [42] S. Khashe, B. Becerik-Gerber, G. Lucas, J. Gratch, Persuasive Effects of Immersion [66] D. Paes, J. Irizarry, A Usability Study of an Immersive Virtual Reality Platform for in Virtual Environments for Measuring Pro-Environmental Behaviors. Proceedings Building Design Review: Considerations on Human Factors and User Interface. of the 35th International Symposium on Automation and Robotics in Construction Proceedings of the Construction Research Congress (CRC 2018), New Orleans, (ISARC 2018), Berlin, 2018, pp. 1205–1211, https://doi.org/10.22260/ 2018, https://doi.org/10.1061/9780784481264.041. ISARC2018/0167. [67] D. Paes, J. Irizarry, The Relevance of Visual Cues in Immersive Environments: Does [43] L. Khemlani, A. Timerman, B. Beene, Y. Kalay, Semantically Rich Building Pictorial Realism Matter? ASCE International Conference on Computing in Civil Representation. Proceeding of the 16th Association for Computer Aided Design in Engineering (i3CE 2019), Atlanta, 2019, pp. 25–31, https://doi.org/10.1061/ Architecture Conference (ACADIA), Cincinnati, 1997, pp. 207–228. ISBN: 9780784482421.004. 1880250063. [68] D. Paes, E. Arantes, J. Irizarry, Immersive environment for improving the [44] S.K. Khuu, J. Gordon, K. Balcomb, J. Kim, The perception of three-dimensional understanding of architectural 3D models: comparing user spatial perception cast-shadow structure is dependent on visual awareness, J. Vis. 14 (25) (2014), between immersive and traditional virtual reality systems, Autom. Constr. 84 https://doi.org/10.1167/14.3.25. (2017) 292–303, https://doi.org/10.1016/j.autcon.2017.09.016. [45] M.J. Kim, X. Wang, P.E.D. Love, H. Li, S.C. Kang, Virtual reality for the built [69] S.E. Palmer, Visual Perception of Objects. Handbook of Psychology: Experimental environment: a critical review of recent advances, J. Inf. Technol. Constr. (ITcon) Psychology, John Wiley & Sons, 2003, https://doi.org/10.1002/0471264385. 18 (2013) 279–304 (ISSN: 1874-4753), http://www.itcon.org/2013/14. wei0407. [46] K. Kimura, J.F. Reichert, A. Olson, O.R. Pouya, X. Wang, Z. Moussavi, D.M. Kelly, [70] M.A. Pourhoseingholi, A.R. Baghestani, M. Vahedi, How to control confounding Orientation in virtual reality does not fully measure up to the real-world, Sci. Rep. effects by statistical analysis, Gastroenterol. Hepatol. Bed Bench 5 (2) (2012) 7 (1) (2017) 1–8, https://doi.org/10.1038/s41598-017-18289-8. 79–83, https://doi.org/10.22037/ghfbb.v5i2.246 (PMID: 24834204; PMCID: [47] C.A. Kelsey, Detection of visual information, in: W.R. Hendee, P. Wells (Eds.), The PMC4017459; ISSN: 2008-2258), https://journals.sbmu.ac.ir/ghfbb/index.php/ Perception of Visual Information, Springer, New York, 1993, pp. 30–51, https:// ghfbb/article/view/246. doi.org/10.1007/978-1-4757-6769-8_2. [71] R.S. Renner, B.M. Velichkovsky, J.R. Helmert, The perception of egocentric [48] S.M. Kosslyn, Image and Brain: The Resolution of the Imagery Debate, MIT Press, distances in virtual environments – a review, ACM Comput. Surv. 46 (2) (2013), Cambridge, 1994. ISBN: 0-262-11184-5. https://doi.org/10.1145/2543581.2543590. [49] R.J. Kovac, The validation of selected spatial ability tests via correlational [72] R. Ruschel, F. Fracaroli, T. Silva, Realidade Virtual Exploratoria´ utilizando-se o assessment and analysis of user-processing strategy, Educ. Res. Q. 13 (2) (1989) Quicktime VR: um estudo sobre a percepçao˜ do ambiente simulado para avaliaçao˜ 26–34. de projeto (Exploratory Virtual Reality using Quicktime VR: a study on the perception of [50] S.F. Kuliga, T. Thrash, R. Dalton, C. Holscher,¨ Virtual reality as an empirical the simulated environment for design assessment). Proceedings of the 2nd Seminar on research tool – exploring user experience in a real building and a corresponding Information Technology and Communication in Civil Construction (TIC 2005), virtual model, Comput. Environ. Urban. Syst. 54 (2015) 363–375, https://doi.org/ EPUSP, Sao Paulo, 2005, pp. 1–10. 10.1016/j.compenvurbsys.2015.09.006. [73] R. Sacks, A. Perlman, R. Barak, Construction safety training using immersive [51] S. Kumar, M. Hedrick, C. Wiacek, J.I. Messner, Developing an experienced-based virtual reality, Constr. Manag. Econ. 31 (9) (2013) 1005–1017, https://doi.org/ design review application for healthcare facilities using a 3D game engine, J. Inf. 10.1080/01446193.2013.828844. Technol. Constr. (ITcon) 16 (2011) 85–104 (ISSN: 1874-4753), http://www.itcon. [74] M.A. Schnabel, T. Kvan, Spatial understanding in immersive virtual environments, org/2011/6. Int. J. Build. Comput. 1 (4) (2003) 435–448, https://doi.org/10.1260/ [52] J. Lazar, J.H. Feng, H. Hochheiser, Research Methods in Human-Computer 147807703773633455. Interaction, 2nd edition, Morgan Kaufmann, Amsterdam, 2017. ISBN: 978-0-12- [75] M.J. Sinai, W.K. Krebs, R.P. Darken, J.H. Rowland, J.S. McCarley, Egocentric 805390-4. distance perception in a virtual environment using a perceptual matching task, [53] R.M. Leicht, S. Kumar, M. Abdelkarim, J. Messner, Gaining End User Involvement Proc. Hum. Factors Ergon. Soc. Annu. Meet. 43 (22) (1999) 1256–1260, https:// Through Virtual Reality Mock-Ups: A Medical Facility Case Study. Proceedings of doi.org/10.1177/154193129904302219. the 27th International Conference CIB W78 - Applications of IT in the AEC [76] M. Slater, Measuring presence: a response to the Witmer and Singer presence Industry, Cairo, ISSN: 2706-656. Available at: https://itc.scix.net/pdfs/w78-201 questionnaire, Presence 8 (5) (1999) 560–565, https://doi.org/10.1162/ 0-143.pdf, 2010. 105474699566477. [54] R. Lovreglio, V. Gonzalez, Z. Feng, R. Amor, M. Spearpoint, J. Thomas, M. Trotter, [77] M. Slater, S. Wilbur, A framework for immersive virtual environments (FIVE): R. Sacks, Prototyping virtual reality serious games for building earthquake speculations on the role of presence in virtual environments, Presence 6 (6) (1997) preparedness: the Auckland City Hospital case study, Adv. Eng. Inform. 38 (2018) 603–616, https://doi.org/10.1162/pres.1997.6.6.603. 670–682, https://doi.org/10.1016/j.aei.2018.08.018. [78] H. Sopher, D. Fisher-Gewirtzman, Y. Kalay, Use of Immersive Virtual Environment [55] J. Lucas, Immersive VR in the construction classroom to increase student in the Design Studio: An Assessment Model. Proceedings of the Education and understanding of sequence, assembly, and space of wood frame construction, J. Inf. Research in Computer Aided Architectural Design in Europe Conference (eCAADe), Technol. Constr. (ITcon) 23 (2018) 179–194 (ISSN: 1874-4753), https://www.itco Lodz, Poland, 2018, pp. 853–862. Available at, http://papers.cumincad.org/data n.org/2018/9. /works/att/ecaade2018_107.pdf. [56] M.L. Maher, Designers and Collaborative Virtual Environments. Collaborative [79] H. Sopher, Y. Kalay, D. Fisher-Gewirtzman, Why Immersive? Using an Immersive Design in Virtual Environments, Intelligent Systems, Control and Automation: Virtual Environment in Architectural Education. Proceedings of the Education and Science and Engineering, 2011, pp. 3–15, https://doi.org/10.1007/978-94-007- Research in Computer Aided Architectural Design in Europe Conference (eCAADe), 0605-7_1. Rome, 2017, pp. 313–322. ISBN: 978-94-91207-12-9. Available at: http://papers. [57] T. Majumdar, M.A. Fischer, B.R. Schwegler, Conceptual Design Review with a cumincad.org/data/works/att/ecaade2017_215.pdf. Virtual Reality Mock-Up Model. Proceedings of the Joint International Conference [80] K.M. Stanney, R.R. Mourant, R.S. Kennedy, Human factors issues in virtual on Computing and Decision Making in Civil and Building Engineering, Montreal,´ environments: a review of the literature, Presence 7 (4) (1998) 327–351, https:// Canada, 2006, pp. 2902–2911. ISBN: 2-921145-58-8. doi.org/10.1162/105474698565767. [58] K. Maldovan, J. Messner, Determining the Effects of Immersive Environments on [81] J. Steuer, Defining virtual reality: dimensions determining telepresence, Decision Making in the AEC Industry. Proceedings of the Joint International J. Commun. 42 (4) (1992) 73–93, https://doi.org/10.1111/j.1460-2466.1992. Conference on Computing and Decision Making in Civil and Building Engineering, tb00812.x. Montreal,´ Canada, 2006, pp. 2762–2770. ISBN: 2-921145-58-8. [82] M. Szumilas, Explaining odds ratios, J. Can. Acad. Child Adolesc. Psychiatry 19 (3) [59] K. Maldovan, J. Messner, M. Faddoul, Framework for Reviewing Mockups in an (2010) 227–229. ISSN: 17198429. Available at: https://www.ncbi.nlm.nih.gov/p Immersive Environment. Proceedings of the 6th International Conference on mc/articles/PMC2938757/. Construction Applications of Virtual Reality (CONVR 2006), Orlando, United [83] V. Thompson, J. Campbell, A power struggle: between- vs. within-subjects designs States, Available at: http://itc.scix.net/pdfs/6d87.content.04272.pdf, 2006. in deductive reasoning research, Psychologia 47 (4) (2004) 277–296, https://doi. [60] D. Marr, Vision: A Computational Investigation into the Human Representation and org/10.2117/psysoc.2004.277. Processing of Visual Information, W. H. Freeman, New York, 1982. ISBN: [84] W.B. Thompson, P. Willemsen, A.A. Gooch, S.H. Creem-Regehr, J.M. Loomis, A. 0716715678. C. Beall, Does the quality of the computer graphics matter when judging distances [61] A.M. Norcia, H.E. Gerhard, Development of three-dimensional perception in in visually immersive environments? Presence 13 (5) (2004) 560–571, https://doi. human infants, Annu. Rev. Vis. Sci. 1 (2015) 569–594, https://doi.org/10.1146/ org/10.1162/1054746042545292. annurev-vision-082114-035835. [85] W. Tizani, Collaborative Design in Virtual Environments at Conceptual Stage. [62] A. Okeil, Hybrid design environments: immersive and non-immersive building Collaborative Design in Virtual Environments, Intelligent Systems, Control and design, J. Inf. Technol. Constr. (ITcon) 15 (2010) 202–216 (ISSN: 1874-4753), Automation: Science and Engineering, 2011, pp. 67–76, https://doi.org/10.1007/ https://www.itcon.org/paper/2010/16. 978-94-007-0605-7_6. [63] M. Oren, P. Carlson, S. Gilbert, J.M. Vance, Puzzle Assembly Training: Real World [86] M. Usoh, E. Catena, S. Arman, M. Slater, Using presence questionnaires in reality, vs. Virtual Environment. Proceedings of the IEEE Virtual Reality Conference, Costa Presence 9 (5) (2000) 497–503, https://doi.org/10.1162/105474600566989. Mesa, United States, 2012, pp. 27–30, https://doi.org/10.1109/VR.2012.6180873. [87] Y. Wang, X. Wang, J. Wang, P. Yung, G. Jun, Engagement of facilities management [64] G. Ozcelik, B. Becerik-Gerber, Benchmarking thermoception in virtual in design stage through BIM: framework and a case study, Adv. Civ. Eng. 2013 environments to physical environments for understanding human-building (2013), https://doi.org/10.1155/2013/189105, 8 pages. interactions, Adv. Eng. Inform. 36 (2018) 254–263, https://doi.org/10.1016/j. aei.2018.04.008.

17 D. Paes et al. Automation in Construction 130 (2021) 103849

[88] J. Wann, M. Mon-Williams, What does virtual reality NEED?: human factors issues [93] X. Yang, S. Ergan, Design and evaluation of an integrated visualization platform to in the design of three-dimensional computer environments, Int. J. Hum. Comput. support corrective maintenance of HVAC problem–related work orders, J. Comput. Studies 44 (6) (1996) 829–847, https://doi.org/10.1006/ijhc.1996.0035. Civ. Eng. 30 (3) (2015), https://doi.org/10.1061/(ASCE)CP.1943-5487.0000510. [89] B.G. Witmer, W.J. Sadowski, Nonvisually guided locomotion to a previously [94] S.Y. Yoon, Psychometric Properties of the Revised Purdue Spatial Visualization viewed target in real and virtual environments, Hum. Factors 40 (3) (1998) Tests: Visualization of Rotations (The Revised PSVT:R), Doctoral Dissertation, 478–488, https://doi.org/10.1518/001872098779591340. Purdue University, West Lafayette, United States, 2011. Available at: https://docs. [90] B.G. Witmer, M.J. Singer, Measuring presence in virtual environments: a presence lib.purdue.edu/dissertations/AAI3480934. questionnaire, Presence 7 (3) (1998) 225–240, https://doi.org/10.1162/ [95] C. Ziemer, J.M. Plumert, J.F. Cremer, J.K. Kearney, Estimating distance in real and 105474698565686. virtual environments: does order make a difference? Atten. Percept. Psychophys. [91] H. Xie, M.E. Tudoreanu, W. Shi, Development of a Virtual Reality Safety-Training 71 (5) (2009) 1095–1106, https://doi.org/10.3758/APP.71.5.1096. System for Construction Workers. Proceedings of the 6th International Conference [96] N. Zikic, Evaluating Relative Impact of VR Components Screen Size, Stereoscopy on Construction Applications of Virtual Reality (CONVR 2006), Orlando, Available and Field of View on Spatial Comprehension and Presence in Architecture (No. 53). at: http://itc.scix.net/data/works/att/ff9b.content.00092.pdf, 2006. Retrieved from The Pennsylvania State University, Computer Integrated [92] N. Yabuki, Impact of Collaborative Virtual Environments on Design Process. Construction (CIC) Research Program, Available at: http://www.pennstatecic.org/ Collaborative Design in Virtual Environments, Intelligent Systems, Control and uploads/5/1/2/1/51219339/tr_053_zikic_2007_vr_in_arch.pdf, 2007. Automation: Science and Engineering, 2011, pp. 103–110, https://doi.org/ 10.1007/978-94-007-0605-7_9.

18