Physically Based Soap Bubble Synthesis for VR

applied sciences

Article Physically Based Soap Bubble Synthesis for VR

Sangwook Yoo, Cheongho Lee and Seongah Chin *

Division of Media Software, Sungkyul University, XICOM LAB, Anyang 430742, Korea; [email protected] (S.Y.); [email protected] (C.L.) * Correspondence: [email protected]

Abstract: To experience a real soap bubble show, materials and tools are required, as are skilled performers who produce the show. However, in a virtual space where spatial and temporal constraints do not exist, bubble art can be performed without real materials and tools to give a sense of immersion. For this, the realistic expression of soap bubbles is an interesting topic for virtual reality (VR). However, the current performance of VR soap bubbles is not satisfying the high expectations of users. Therefore, in this study, we propose a physically based approach for reproducing the shape of the bubble by calculating the measured parameters required for bubble modeling and the physical motion of bubbles. In addition, we applied the change in the ﬂow of the surface of the soap bubble measured in practice to the VR rendering. To improve users’ VR experience, we propose that they should experience a bubble show in a VR HMD (Head Mounted Display) environment.

Keywords: soap bubble; soap bubble modeling; motion vector texture; VR

1. Introduction With the recent advances in computer graphics technology, the extended reality (XR) market, which encompasses virtual reality (VR) and augmented reality (AR), is rapidly growing, and various content is being produced. In recent years, even artificial intelligence Citation: Yoo, S.; Lee, C.; Chin, S. (AI) has been applied to VR applications and has been used in various fields such as Physically Based Soap Bubble tourism, leisure, education, games, and health care fields [1]. Orchestra performances and Synthesis for VR. Appl. Sci. 2021, 11, music videos by singers are provided in VR, and the types of application that are viewed 3090. https://doi.org/10.3390/ are continually growing. However, to watch an immersive soap bubble show, we need real app11073090 materials such as soap bubbles and bubble tools along with a special concert hall, and it must be accompanied by experienced performers who produce it. Therefore, in this work, Received: 9 March 2021 we propose a VR bubble show framework that has no space-time constraints and can be Accepted: 29 March 2021 directed without special bubble art materials and equipment. In addition, at a time when Published: 31 March 2021 outdoor activities are restricted due to COVID-19 Pandemic, soap bubble can be considered a timely technology to experience performance indoors as well. Publisher’s Note: MDPI stays neutral To synthesize a soap bubble show in VR, we need to analyze and model the actual with regard to jurisdictional claims in physical shape and movement of soap bubbles. However, modeling soap bubbles in published maps and institutional affil- a VR environment requires too much computation and there are hardware limitations. iations. Nevertheless, the realistic expression of soap bubbles is very important and it must satisfy users’ high expectations. To realize a soap bubble show in VR, we first need to understand the basic properties of soap bubbles. Soap bubbles are very light in the form of a thin layer of soap that surrounds the air, with large changes caused even with a small force [2]. The Copyright: © 2021 by the authors. soap bubbles used in a bubble show are more stable than typical soap bubbles because Licensee MDPI, Basel, Switzerland. they use a liquid containing surfactant ingredients that stops the bubble from bursting This article is an open access article easily [3]. In addition, many physical laws apply to a single soap bubble and determine its distributed under the terms and movement patterns. In computer graphics, soap bubbles were realistically simulated as conditions of the Creative Commons clusters of many soap bubbles using vortex sheets [4]. As hardware capable of real-time Attribution (CC BY) license (https:// rendering was developed, motion recognition was used to simulate the generation and creativecommons.org/licenses/by/ bursting of soap bubbles [5]. A study even examined color changes in bubbles due to light 4.0/).

Appl. Sci. 2021, 11, 3090. https://doi.org/10.3390/app11073090 https://www.mdpi.com/journal/applsci Appl. Sci. 2021, 11, 3090 2 of 17

cancellation and interference [6]. To the best of our knowledge, no study has used soap bubble modeling to determine the size of a soap bubble based on actual parameters or applied the laws of physics to bubbles that are to be used in VR content, nor has a study produced a rendering model to which environmental variables can be applied. In this study, we propose a physically based soap bubble synthesis that can be experienced in a virtual reality environment. To this end, we analyzed measured soap bubble data and optimized the motion vector texture to express the velocity of ﬂuid ﬂow. In addition, for optimized modeling, the shape and motion of soap bubbles were measured and parameters describing the physical characteristic were extracted. We measured the movement path, internal and external pressure difference, damping force, volume, mass, cross-sectional area, air resistance acceleration, and fall acceleration of soap bubbles. Finally, we implemented a soap bubble experience in VR by applying the physics-based rendering technology. The contributions of this study are summarized as follows: We created a soap bubble model by measuring several physical variables of soap bubbles; we developed a soap bubble modeling method optimized for VR by applying the physical variables required for soap bubble modeling and rendering. Finally, we provided tools that can be experienced in VR environments and tools for creating customized hand gestures such as clap, grab, fanning, bouncing and piercing.

2. Related Work We reviewed previous studies with an interest in art culture and various aspects of engineering related to rendering and modeling of soap bubbles and their simulations. A previous study reported the thin film interference of soap bubbles [7]. This study measured soap bubble thickness with color matching by measuring the thickness at various points using the principle of color perception and by utilizing the thin film interference of soap bubbles. This study contributed to our understanding by showing how the surface color reflected through the thin film interference depended on the thickness of the soap film according to the viewing distance and angle. A study on the surface tension of soap bubbles found that the thickness of flowing soap films influenced the strength of the surface tension [8]. Real-time rendering of soap bubbles that measured light interference showed how the surface tension could change in real time [9]. This study explained that external forces and the bubble’s surface thickness are the main causes of variation in surface tension. Another study examined the formation and break-up of fluid bubbles, and found that this process contributed to changes in the thickness and shape of flowing soap bubbles [10]. A high-speed visualization of soap bubble blowing and image-processing-based analysis of pinch-off dynamics captured the moment when soap bubbles separated [11]. A theoretical and experimental study of the stability of a soap film spanning a flexible loop [12] proposed that the stability of soap bubbles was generated by twisting or bending the tool on the same dynamic tool rather than a static tool. Another study was conducted on soap bubbles and the surrounding environment [13]. This study reported a method that involved spreading soap bubbles on dry and wet surfaces, and examined the friction and bursting of soap bubbles according to the surrounding environment. The rendering conditions of soap bubbles in terms of reflection and refraction, as well as the wavelength of the soap bubble color, has also been studied [14]. A model for soap film dynamics with evolving thickness [15] presented a new algorithm to render a soap film. This study analyzed the changing thickness as the soap bubble flowed, which contributed to changes at the bubble’s surface and to the geometric shape. Another study reported an irregular flow rate measurement method for soap films [16]. The effect of chemo-mechanical simulation of soap film flows on spherical bubbles proposed rendering based on lubrication theory [17]. This study reproduced the free flow of the soap film and change of flow due to external forces such as gravity and air flow. Appl. Sci. 2021, 11, x FOR PEER REVIEW 3 of 18 Appl. Sci. 2021, 11, x FOR PEER REVIEW 3 of 18

The effect of chemo-mechanical simulation of soap film flows on spherical bubbles pro- Appl. Sci. 2021, 11, 3090 3 of 17 posedThe effect rendering of chemo-mechanical based on lubrication simulation theory of [17]. soap This film study flows reproduced on spherical the bubbles free flow pro- of theposed soap rendering film and based change on of lubrication flow due totheory external [17]. forces This studysuch as reproduced gravity and the air free flow. flow of the soap film and change of flow due to external forces such as gravity and air flow. 3.3. SoapSoap BubbleBubble ModelingModeling 3. Soap Bubble Modeling MostMost existingexisting soapsoap bubblebubble modelsmodels areare beingbeing usedused withoutwithout consideringconsidering thethe sizesize oror standardstandardMost of of existing the the soapsoap soap bubblebubble bubble handling handling models tool. tool. are However,Ho beinwever,g used the the size sizewithout and and shape shape considering of of the the soap soap the bubble bubblesize or clearlyclearlystandard dependsdepends of the soap onon thethe bubble sizesize of ofhandling thethe tooltool tool. used.used. Ho Therefore,Therefore,wever, the toto size makemake anda a soapshape soapbubble bubbleof the soapmodel model bubble that that isisclearly closercloser depends toto reality,reality, on it itthe is is necessary sizenecessary of the to toolto measure meas used.ure Therefore, the the exact exact value tovalue make of of physicala physical soap bubble parameters. parameters. model that ToTo thisthisis closer end,end, wetowe reality, conductedconducted it is an necessaryan experimentexperiment to meas toto measuremeasureure the theexactthe diameter,diameter, value of shape, shape,physical and and parameters. movement movement ofTo of bubbles.bubbles.this end, WhenWhenwe conducted aa soapsoap bubblebubble an experiment isis created,created, to themeasurethe threethree importanttheimportant diameter, parametersparameters shape, and toto movement considerconsider areare of thethebubbles. verticalvertical When diameterdiameter a soap of of the bubblethe bubble,bubble, is created, the the separationsepara the tionthree position, position, important and and parameters the the diameter diameter to of ofconsider the the bubble bubble are afterafterthe vertical separation.separation. diameter of the bubble, the separation position, and the diameter of the bubble after separation. 3.1.3.1. MeasurementMeasurement ofof SoapSoap BubbleBubble EnvironmentalEnvironmental VariablesVariables 3.1. Measurement of Soap Bubble Environmental Variables InIn thethe experiment,experiment, soapsoap bubblesbubbles werewere blownblown directlydirectly withwith aa blackblack backgroundbackground andand ﬁlmedfilmedIn using usingthe experiment, 1080p1080p resolutionresolution soap bubbles slow-motionslow-motion were blow video.video.n directly TheThe featurefeature with valuevalue a black ofof thebackgroundthe soapsoap bubblebubble and waswasfilmed extractedextracted using 1080p inin cmcm using resolutionusing thethe scalescale slow-motion ratioratio betweenbetween video. thethe The cameracamera feature andand value thethe object objectof the sizesize soap fromfrom bubble thethe stillstillwas image imageextracted obtainedobtained in cm fromfrom using thethe the capturedcaptured scale ratio video. video. between FigureFigure the1 1shows showscamera some some and of theof the the object still still images sizeimages from used. used. the FigureFigurestill image1 a1a is is obtained a a small small straw strawfrom withthe with captured a a diameter diameter video. of of 0.7 Figure0.7 cm, cm,and 1 and shows Figure Figure some1b 1b is of is a the largea large still straw imagesstraw with with used. a a diameterdiameterFigure 1a of ofis 1.31.3a small cm.cm. There Therestraw are arewith threethree a diameter keykey parametersparameters of 0.7 cm, whenwhen and theFigurethe soapsoap 1b bubblebubble is a large isis created.created. straw with First,First, a the change in the vertical diameter of the soap bubble, second, the change in the separated thediameter change of in 1.3 the cm. vertical There diameter are three ofkey the parameters soap bubble, when second, the soap the changebubble inis created.the separated First, position, when the soap bubble is completely separated, and third, the change in the soap position,the change when in the the vertical soap bubble diameter is completely of the soap separated, bubble, second, and third, the change the change in the in separated the soap bubble diameter after separation. Figure2 is a 3D model of soap bubble. Vertical diameter, bubbleposition, diameter when the after soap separation. bubble is Figure completely 2 is a 3Dseparated, model of and soap third, bubble. the change Vertical in diameter, the soap separated position, and soap bubble diameter can be checked in the soap bubble. separatedbubble diameter position, after and separation. soap bubble Figure diameter 2 is a 3D can model be ch eckedof soap in bubble. the soap Vertical bubble. diameter, separated position, and soap bubble diameter can be checked in the soap bubble.

FigureFigure 1.1. SoapSoap bubble photos taken: ( (aa)) A A small small straw straw with with a adiameter diameter of of 0.7 0.7 cm, cm, and and (b ()b A) Alarge large strawFigure with 1. Soap a diameter bubble photosof 1.3 cm. taken: (a) A small straw with a diameter of 0.7 cm, and (b) A large straw with a diameter of 1.3 cm. straw with a diameter of 1.3 cm.

Figure 2. Pre-stage parameters. FigureFigure 2.2.Pre-stage Pre-stage parameters.parameters. 3.2. Measuring the Vertical Diameter and Separated Positon of Soap Bubbles 3.2.3.2. MeasuringMeasuring thethe VerticalVertical DiameterDiameter andand SeparatedSeparated PositonPositonof of Soap Soap Bubbles Bubbles The vertical diameter and separated position of soap bubbles were determined by the shapeTheThe verticalvertical and size diameter diameter of the tool and and blowing separated separated the position soappositi bubble.on of soapof soap Therefore, bubbles bubbles were in werethis determined study, determined 30 byexper- the by shapeimentsthe shape and were sizeand conducted ofsize the of tool the with blowingtool straws blowing the with soap the a bubble.soap diameter bubble. Therefore, of 1.3Therefore, cm in and this strawsin study, this study,with 30 experiments a 30diameter exper- wereiments conducted were conducted with straws with with straws a diameter with a diameter of 1.3 cm andof 1.3 straws cm and with straws a diameter with a of diameter0.7 cm. Figure3a shows the vertical diameter value extracted from the experiment when the soap bubble was blown. When using a 1.3 cm straw, the average diameter was 1.43 cm (SD: 0.18 cm), while for the 0.7 cm straw the average diameter was 0.91 cm (SD: 0.16 cm). The Appl. Sci. 2021, 11, x FOR PEER REVIEW 4 of 18 Appl. Sci. 2021, 11, x FOR PEER REVIEW 4 of 18

Appl. Sci. 2021, 11, 3090 of 0.7 cm. Figure 3a shows the vertical diameter value extracted from the experiment when4 of 17 of 0.7 cm. Figure 3a shows the vertical diameter value extracted from the experiment when the soap bubble was blown. When using a 1.3 cm straw, the average diameter was 1.43 the soap bubble was blown. When using a 1.3 cm straw, the average diameter was 1.43 cm (SD: 0.18 cm), while for the 0.7 cm straw the average diameter was 0.91 cm (SD: 0.16 cm (SD: 0.18 cm), while for the 0.7 cm straw the average diameter was 0.91 cm (SD: 0.16 cm). The vertical diameter ranged from 1.24 to 1.6 cm for the 1.3 cm straw, and from 0.76 cm).vertical The diameter vertical diameter ranged from ranged 1.24 from to 1.6 1.24 cm forto 1.6 the cm 1.3 for cm the straw, 1.3 andcm straw, from 0.76and tofrom 1.07 0.76 cm to 1.07 cm for the 0.7 cm straw. tofor 1.07 the cm 0.7 cmfor the straw. 0.7 cm straw. 2.5 20 2.5 1.3cm 20 1.3cm 2 0.7cm1.3cm 0.7cm1.3cm 15 0.7cm 2 0.7cm 15 1.5 1.5 10 1 10 1 5 0.5 5 0.5 0

Vertical Diameter Vertical Diameter (cm) 0 Separated Position (cm) Position Separated 0 Vertical Diameter Vertical Diameter (cm) 10 3 5 7 9 11 13 15 17 19 21 23 25 27 29 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 1 3 5 7Sample 9 11 13Index 15 17 19 21 23 25 27 29 (cm) Position Separated 1 3 5 7Sample 9 11 13 Index 15 17 19 21 23 25 27 29 Sample Index (Samplea) Index (b) (a) (b) Figure 3. (a) Vertical diameter of bubbles blown using a 1.3 and 0.7 cm straw and (b) Separated positionFigure of bubbles3. (a(a) )Vertical Vertical using diametera diameter 1.3 and of0.7 of bubbles cm bubbles straw. blown blown using using a 1.3 a 1.3 and and 0.7 0.7 cm cm straw straw and and (b) (Separatedb) Separated position of bubbles using a 1.3 and 0.7 cm straw. Figure 3b shows the value of the separated position for bubbles. When using a 1.3 cm Figure 3b shows the value of the separated position for bubbles. When using a 1.3 cm straw, theFigure average3b showsseparated the valueposition of thewas separated 10.1 cm (SD: position 1.9 cm), for bubbles.and with When the 0.7 using cm straw a 1.3 cm straw, the average separated position was 10.1 cm (SD: 1.9 cm), and with the 0.7 cm straw the averagestraw, the was average 8.39 cm separated (SD: 2.7 cm). position The wasseparated 10.1 cm position (SD: 1.9 ranged cm), and from with 8.2 theto 12 0.7 cm cm for straw the average was 8.39 cm (SD: 2.7 cm). The separated position ranged from 8.2 to 12 cm for the 1.3the cm average straw wasand 8.39from cm 5.73 (SD: to 2.711.05 cm). cm The for separatedthe 0.7 cm position straw. ranged from 8.2 to 12 cm for the 1.3 cm straw and from 5.73 to 11.05 cm for the 0.7 cm straw. 3.3. Soap Bubble Diameter 3.3. Soap Soap Bubble Diameter To measure the soap bubble diameter, we conducted an experiment using two cases: To measure the soap bubble diameter, we conducted an experiment using two cases: strong and fast blown bubbles, and weak and slow blown bubbles. First, it was observed strong and fast blown bubbles, and weak and slow blown bubbles. First, it was observed that several small soap bubbles appeared in the case of strong and rapid blowing (Figure that several smallsmall soapsoap bubbles bubbles appeared appeared in in the the case case of of strong strong and and rapid rapid blowing blowing(Figure (Figure4a) . 4a). In the case of weak and slow blowing, one large soap bubble was observed (Figure 4a).In the In casethe case of weak of weak and and slow slow blowing, blowing, one one large large soap soap bubble bubble was was observed observed(Figure (Figure4b) . 4b). The diameter was calculated over time using a machine learning algorithm [6]. The 4b).The The diameter diameter was was calculated calculated over over time time using using a machine a machine learning learning algorithm algorithm [6]. The [6]. soap The soap bubble diameter gradually increased over time. The soap bubble diameter using a soapbubble bubble diameter diameter gradually graduall increasedy increased over time. over Thetime. soap The bubble soap bubble diameter diameter using ausing 1.3 cm a 1.3 cm straw ranged from 1.92 to 3.37 cm, while for the 0.7 cm straw the values ranged 1.3straw cm ranged straw ranged from 1.92 from to 3.371.92 cm,to 3.37 while cm, for while the 0.7 for cm the straw 0.7 cm the straw values the ranged values from ranged 1.32 from 1.32 to 2.21 cm. The error range in the value corresponding to each second shows a fromto 2.21 1.32 cm. to The2.21 errorcm. The range error in range the value in the corresponding value corresponding to each secondto each second shows ashows 0.5 cm a 0.5 cm deviation. 0.5deviation. cm deviation.

5 5 5 1.3cm 5 1.3cm 4 1.3cm 4 1.3cm 4 0.7cm 4 0.7cm 3 0.7cm 3 0.7cm 2 3 2 3 1 2 1 2 (cm) 0 1 (cm) 1 (cm) 0 (cm)

1 0 2 2 0 1.2 1.3 1.7 1 2 2 1.37 1.46 1.56 1.63 1.75 1.86 1.89 1.93 1.2 1.3 1.7 Soap Bubble Diameter Diameter Bubble Soap Soap Bubble Diameter Diameter Bubble Soap 1.37 1.46 1.56 1.63 1.75 1.86 1.89 1.93 Soap Bubble Diameter Diameter Bubble Soap Soap Bubble Diameter Diameter Bubble Soap Time (sec.) Time (sec.) Time (sec.) Time (sec.) (a) (b) (a) (b) FigureFigure 4. (a) 4. Soap(a) Soap bubble bubble diameter diameter measured measured for strong for strong and andfast fastblown blown bubbles bubbles using using 1.3 1.3and and 0.7 0.7 cm Figure 4. (a) Soap bubble diameter measured for strong and fast blown bubbles using 1.3 and 0.7 cm straws.straws. (b (b) )Soap Soap bubble bubble diameter diameter measured measured for for weak weak and and slow slow blown blown bubbles bubbles using using 1.3 1.3 and and 0.7 0.7 cm cm straws.cm straws. (b) Soap bubble diameter measured for weak and slow blown bubbles using 1.3 and 0.7 cmstraws. straws. 3.4. Soap3.4. SoapBubble Bubble Parameters Parameters and Tool and ToolModel Model 3.4. Soap Bubble Parameters and Tool Model The Thesoap soap bubble bubble data data measured measured through through Section Section 3.2 are 3.2 usedare used to create to create it in itinitial in initial 3D 3D The soap bubble data measured through Section 3.2 are used to create it in initial 3D MAXMAX in Figure in Figure 5a 5anda and then then apply apply it itin in Unity. Unity. As As for for the the procedure procedure order, thethe 3D3D modelmodel was MAX in Figure 5a and then apply it in Unity. As for the procedure order, the 3D model was firstfirst createdcreated fromfrom the the directly directly measured measured photos, photos, videos, videos, and and data, data, and and the the starting starting shape was first created from the directly measured photos, videos, and data, and the starting shapeand and shape shape of of the the soap soap bubble bubble were were created created using using the the 3D 3D MAX MAX tool, tool, and and the the morphing morph- was shape and shape of the soap bubble were created using the 3D MAX tool, and the morphing wascomputed computed through through the final the final animation. animation. After After that, that, the created the created FBX (Filmbox)FBX (Filmbox) file is file loaded ingin Unity was computed by coding, through created the to befinal called animation. out at theAfter desired that, the size created and speed, FBX (Filmbox) and the soap file bubbles are separated and moved according to the laws of physics. Figure5b is a schematic diagram showing how to design a soap bubble before making it. The soap bubble model of the straw was designed to be made by applying the point that the soap bubble pops out Appl. Sci. 2021, 11, x FOR PEER REVIEW 5 of 18

is loaded in Unity by coding, created to be called out at the desired size and speed, and Appl. Sci. 2021, 11, 3090 the soap bubbles are separated and moved according to the laws of physics. Figure5 of5b 17 is a schematic diagram showing how to design a soap bubble before making it. The soap bubble model of the straw was designed to be made by applying the point that the soap fasterbubble than pops when out usingfaster otherthan when tools, andusing is other divided tools, into and a cylindrical is divided part into that a cylindrical is blown outpart andthat a is soap blown bubble out and part a thatsoap becomes bubble part a circular that becomes model. a The circular 0.7 cm model. straw The and 0.7 the cm 1.3 straw cm strawsand the differ 1.3 cm in straws size but differ used in the size same but used model. the Sincesame themodel. soap Since bubble the modelsoap bubble of the model tube comesof the outtube slowly, comes unlike out slowly, the straw unlike model, the straw the distinction model, the between distinction the blown between part the and blown the soappart bubbleand the part soap is bubble not necessarily part is not distinctive. necessarily Therefore, distinctive. the Therefore, tube model the was tube designed model was to producedesigned one to modelproduce in one which model the in two which were the combined. two were The combined. hoop soap The bubble hoop soap model bubble was notmodel tubular, was butnot rathertubular, a thinbut rather circular a thin type, circ soular it was type, constructed so it was asconstructed a model that as a occurs model whenthat occurs the stick when is swung. the stick In is reality, swung. the In hoop reality, is swungthe hoop in is various swung directions, in various but directions, in this study,but in thethis case study, where the thecase hoop where is the swung hoop in is a swung straight in line a straight was speciﬁed line was and specified produced. and Figureproduced.6 shows Figure the 6 model shows that the producedmodel that soap prod bubblesuced soap using bubbles the measured using the or measured predicted or values.predicted It can values. be seen It can that be it seen comes that out it similar comes toout a realsimilar soap to bubble. a real soap It was bubble. ﬁlmed It over was timefilmed as soapover bubblestime as soap were bubbles blown out were from blown top toout bottom. from top to bottom.

(a)

(b) Appl. Sci. 2021, 11, x FOR PEER REVIEW 6 of 18 FigureFigure 5. 5.Initial Initial 3D 3D MAX MAX models models ( a(a),), and and parameter parameter design design for for each each soap soap bubble bubble tool tool ( b(b).).

Figure 6. Pre-stage simulation results for Straw 2 from Figure 5. Figure 6. Pre-stage simulation results for Straw 2 from Figure5. 3.5. Movement Path of Soap Bubbles Soap bubbles that are blown out move under the influence of physical laws such as wind. However, in this study, the movement of soap bubbles was expressed based on the data measured during the experiment. In the experiment, the coordinates of each movement route were extracted by photographing the bubble blowing from the side. Since it was inefficient to extract all coordinates, only a fixed number of coordinates per path were extracted, and the path between each set of coordinates was determined using the Bezier spline equation so that the movement of the soap bubble was not linear but moved in a curved line. The speed of a soap bubble moving along its path was determined by the strength of the wind when it was blown out and the volume in its separated state. The first soap bubble that came out was fast because of the power of wind. However, because the air resistance acceleration due to the light mass is large, the speed decreases rapidly [18]. The formula for acceleration representing the velocity of a bubble is as follows:

𝑎 = 𝛾/𝑚 −𝐹 (1)

where 𝛾 is the external force acting on the soap bubble, which includes gravity, m is the mass, and 𝐹 is the air resistance acceleration. As the air resistance acceleration increases, the speed of the soap bubble decreases. The formula for the air resistance acceleration is as follows: 𝐶𝜌𝐴𝑣 𝐹 = (2) 2𝑚 where 𝐶 is the air resistance coefficient, ρ is the density of the bubble, A is the cross-sectional area of the bubble, m is the mass, and v is the velocity. The air resistance coefficient and density depend on the shape of the soap bubble or the concentration of surfactant, but in this study, the air resistance coefficient was assumed to be 0.47 and the density to be 0.00125 g/cm2.

3.6. Force Acting on the Surface of the Soap Bubble When a soap film is stretched, the surface tension of the stretched portion increases. However, applying a change to the surface tension along the surface of the soap bubble is inefficient because the increased computational demand is disproportionately high relative to the improvement in the experience of the user. Therefore, in this study, the surface tension was calculated assuming a constant, and the motion of the surface was expressed by using the pressure difference between the inside and outside of the bubble, which changes according to the diameter of the bubble. The pressure difference between the inside and outside determines the speed at which the soap bubble moves to have the minimum area, and the formula is as follows: Appl. Sci. 2021, 11, 3090 6 of 17

3.5. Movement Path of Soap Bubbles Soap bubbles that are blown out move under the influence of physical laws such as wind. However, in this study, the movement of soap bubbles was expressed based on the data measured during the experiment. In the experiment, the coordinates of each movement route were extracted by photographing the bubble blowing from the side. Since it was inefficient to extract all coordinates, only a fixed number of coordinates per path were extracted, and the path between each set of coordinates was determined using the Bezier spline equation so that the movement of the soap bubble was not linear but moved in a curved line. The speed of a soap bubble moving along its path was determined by the strength of the wind when it was blown out and the volume in its separated state. The first soap bubble that came out was fast because of the power of wind. However, because the air resistance acceleration due to the light mass is large, the speed decreases rapidly [18]. The formula for acceleration representing the velocity of a bubble is as follows:

a = γ/m − Fb (1)

where γ is the external force acting on the soap bubble, which includes gravity, m is the mass, and Fb is the air resistance acceleration. As the air resistance acceleration increases, the speed of the soap bubble decreases. The formula for the air resistance acceleration is as follows: CρAv2 F = (2) b 2m where C is the air resistance coefficient, ρ is the density of the bubble, A is the cross-sectional area of the bubble, m is the mass, and v is the velocity. The air resistance coefficient and density depend on the shape of the soap bubble or the concentration of surfactant, but in this study, the air resistance coefficient was assumed to be 0.47 and the density to be 0.00125 g/cm2.

3.6. Force Acting on the Surface of the Soap Bubble When a soap ﬁlm is stretched, the surface tension of the stretched portion increases. However, applying a change to the surface tension along the surface of the soap bubble is inefﬁcient because the increased computational demand is disproportionately high relative to the improvement in the experience of the user. Therefore, in this study, the surface tension was calculated assuming a constant, and the motion of the surface was expressed by using the pressure difference between the inside and outside of the bubble, which changes according to the diameter of the bubble. The pressure difference between the inside and outside determines the speed at which the soap bubble moves to have the minimum area, and the formula is as follows:

4σ ∆p = (3) d where σ is the assumed surface tension of the bubble and d is the diameter of the bubble produced. The internal and external pressure difference decreases as the diameter of the soap bubble increases. Due to this, as the size of the soap bubble increases, the surface of the soap bubble changes in various ways. When a force acts on the soap bubble, a change occurs on the surface, and the force gradually decreases based on the point of action and is transmitted to the surroundings. As a result, when force is applied to the soap bubble, not only does a part of the surface change, but the entire soap bubble dynamically changes, as does its shape. The surface movement begins, which affects the entire bubble. The damping force according to the distance from the force application point can be calculated as follows: r u(x) = (4) 2π(x − x0) Appl. Sci. 2021, 11, 3090 7 of 17

where r is the external force acting on the bubble, excluding gravity, and x − x0 is the distance from the point of application of the force. The damping force decreases further from the point of action, and when a force is applied to the surface of the soap bubble, the surrounding surface also changes, and the stronger the force applied, the greater the change.

4. Soap Bubble Rendering 4.1. Flow Speed of Soap Film Surface In a previous study, the authors reported measuring the flow speed of a physics- based soap film [19,20]. Based on this, we introduce an improved rendering method that reflects the speed of each section. Figure7a shows the ‘UV Coordinate’ function of UV Mapping [21] that makes 2D textures into 3D models in a shaded graph, which is a node- based shading editor of Unity. UVW is the same coordinate concept as XYZ, and W is not used because it uses a 2D texture. Therefore, U is the X-axis and V is the Y-axis. As a result of separating each channel of UV, it can be seen in Figure7b,c that each channel has a directional value of each axis by pixels from 0 to 255. Using this, the direction of motion can be determined by adjusting the color value of each pixel. In this study, a height map was used to determine the direction of motion. A height map is a 2D texture composed of grayscale and functions as displacement mapping with displacement information. The larger the displacement, the closer it is to white, and the smaller the displacement, the closer it is to black. In this study, Figure7e was produced by dividing the soap bubble into five sections (Figure7d) and optimizing the height map. Finally, a motion vector texture (Figure7f), which can control the direction and speed through an overlay on the UV coordinate with the direction value and the height map with the displacement value, was produced through motion vector texture production. Table1 is a graph comparing motion vector texture and measured data by dividing them into five sections as shown in Figure7d. Each graph shows the rate of change of a pixel as a trend line of the polynomial. The average error accuracy and the root mean square error accuracy were used to evaluate the analysis model of the graph [22]. The average error is the value obtained by calculating the error of the speed in the section where the measured data and the pixel index of the motion vector texture correspond. It is Appl. Sci. 2021, 11, x FOR PEERthe REVIEW square root of the arithmetic mean of the error and means the mutual deviation. Each 9 of 18

graph has a similar shape with a speed value error (relative speed) of around 3 for each section, and the accuracy according to this is shown in Table2. The mean root-square error accuracy was about 97.6%, and the mean error accuracy was about 88.6%, conﬁrming the error accuracy was about 97.6%, and the mean error accuracy was about 88.6%, confirm- reliability of the motion vector texture. ing the reliability of the motion vector texture.

Table 1. Comparison between experimental speed and motion vector texture graph for each section Table 1. Comparison between experimental speed and motion vector texture graph for each sec- of the soaption bubble. of the soap bubble.

SectionSection Graph Graph

1/5

2/5

3/5

4/5

5/5

Appl. Sci. 2021, 11, x FOR PEER REVIEW 9 of 18 Appl. Sci. 2021, 11, x FOR PEER REVIEW 9 of 18 Appl. Sci. 2021, 11, x FOR PEER REVIEW 9 of 18

error accuracy was about 97.6%, and the mean error accuracy was about 88.6%, confirm- errorerror accuracy accuracy was was about about 97.6%, 97.6%, and and the the mean mean error error accuracy accuracy was was about about 88.6%, 88.6%, confirm- confirm- ing the reliability of the motion vector texture. inging the the reliability reliability of of the the motion motion vector vector texture. texture.

Table 1. Comparison between experimental speed and motion vector texture graph for each sec- TableTable 1. 1. Comparison Comparison between between experimental experimental speed speed and and motion motion vector vector texture texture graph graph for for each each sec- sec- tion of the soap bubble. tiontion of of the the soap soap bubble. bubble. Section Graph SectionSection GraphGraph

Appl. Sci. 2021, 11, 3090 8 of 17 1/5 1/51/5

Table 1. Cont.

Section Graph

2/5 2/52/52/5

3/5 3/53/53/5

4/54/5 4/54/5

5/55/5 5/55/5 Appl. Sci. 2021, 11, x FOR PEER REVIEW 10 of 18

AccuracyAccuracy

Table 2. Mean average error accuracy and root mean square error accuracy for each section of the soap bubble. RMSE: Root mean square error; MAE: Mean error accuracy.

Accuracy Section 1/5 2/5 3/5 4/5 5/5 Average RMSE 97.9529% 98.2817% 97.1969% 98.4569% 96.2519% 97.6281% MAE 90.5685% 90.7778% 87.9233% 90.9751% 82.8941% 88.6278%

4.2. Soap Bubble Rendering Parameters Soap bubbles are very light because they are formed of a very thin layer of soap that envelopes the air, and a small force produces a large change in a bubble. When this film receives light, the waves are reflected from the surface and inside. As shown in Figure 8, the color looks different depending on the viewing distance and angle of the viewpoint reflected by interference known as iridescence [23]. In this study, as a result of measuring the change in the flow shape and thickness of the fluid by observing the movement of the soap film, a soap bubble texture that can sample the flow shape, thickness and color values of the fluid was produced [24]. The formula in Equations (5) and (6) for this are as follows [25]:

𝜎 = 𝜂𝑑𝑐𝑜𝑠 𝜃 λ2𝑚 1 𝑚 =0,1,2… (5)

𝜎 = 𝜂𝑑𝑐𝑜𝑠 𝜃 λ2𝑚 𝑚 =0,1,2… (6)

where 𝑑 is the thickness and 𝜂 is the refractive index. A ray of light with wavelength λ enters this thin film. If half the wavelength is an odd multiplication, there will be a bright constructive interference using (5). On the other hand, if half the wavelength is an even multiplication, there will be a dark destructive interference using (6). We used this principle to implement thin film interventions and iridescence.

Figure 8. Principle of thin-film interference.

According to basic theory of physically based rendering [26], the physically based rendering pipeline can be classified into basic color, metallicity, roughness, Fresnel reflectance, refractive index, and reflection of materials. In addition, the movement change, distortion, opacity, and surrounding occlusion can be considered. We implemented the soap Appl. Sci. 2021, 11, 3090 9 of 17

Table 2. Mean average error accuracy and root mean square error accuracy for each section of the soap bubble. RMSE: Root mean square error; MAE: Mean error accuracy.

Accuracy Section 1/5 2/5 3/5 4/5 5/5 Average

Appl. Sci. 2021, 11, x FOR PEER REVIEW RMSE 97.9529% 98.2817% 97.1969% 98.4569% 96.2519% 97.6281%8 of 18 MAE 90.5685% 90.7778% 87.9233% 90.9751% 82.8941% 88.6278%

(a) UV Coordinate (b) UV Coordinate (Ch. Red)

(e) Height map (f) Motion vector texture

FigureFigure 7. 7.PrinciplePrinciple of of motion motion vector vector texture. texture.

Table 1 is a graph comparing motion vector texture and measured data by dividing them into five sections as shown in Figure 7d. Each graph shows the rate of change of a pixel as a trend line of the polynomial. The average error accuracy and the root mean square error accuracy were used to evaluate the analysis model of the graph [22]. The average error is the value obtained by calculating the error of the speed in the section where the measured data and the pixel index of the motion vector texture correspond. It is the square root of the arithmetic mean of the error and means the mutual deviation. Each graph has a similar shape with a speed value error (relative speed) of around 3 for each section, and the accuracy according to this is shown in Table 2. The mean root-square Appl. Sci. 2021, 11, x FOR PEER REVIEW 10 of 18

Accuracy

Table 2. Mean average error accuracy and root mean square error accuracy for each section of the soap bubble. RMSE: Root mean square error; MAE: Mean error accuracy.

Accuracy Section 1/5 2/5 3/5 4/5 5/5 Average RMSE 97.9529% 98.2817% 97.1969% 98.4569% 96.2519% 97.6281% Appl. Sci. 2021, 11, 3090 MAE 90.5685% 90.7778% 87.9233% 90.9751% 82.8941% 88.6278%10 of 17

4.2. Soap Bubble Rendering Parameters Soap bubbles are very light because they are formed of a very thin layer of soap that 4.2. Soap Bubble Rendering Parameters envelopes the air, and a small force produces a large change in a bubble. When this film receivesSoap light, bubbles the arewaves very are light reflected because from they th aree surface formed and of ainside. very thin As shown layer of in soap Figure that 8, envelopesthe color thelooks air, different and a small depending force produces on the vi aewing large changedistance in and a bubble. angle of When the viewpoint this film receivesreflected light, by interference the waves are known reflected as iridescence from the surface[23]. In andthis inside.study, as As a shownresult of in measuring Figure8, thethe color change looks in the different flow shape depending and thickness on the viewing of the fluid distance by observing and angle the of movement the viewpoint of the reflectedsoap film, by a interference soap bubble known texture as that iridescence can sample [23 the]. In flow this shape, study, thickness as a result and of measuringcolor values theof the change fluid in was the produced flow shape [24]. and The thickness formula ofin Equations the fluid by (5) observing and (6) for the this movement are as follows of the soap film, a soap bubble texture that can sample the flow shape, thickness and color [25]: values of the fluid was produced [24]. The formula in Equations (5) and (6) for this are as follows [25]: 𝜎 = 𝜂𝑑𝑐𝑜𝑠 𝜃 λ2𝑚 1 𝑚 =0,1,2… (5) σa = η f ilmd f ilmcos θ λ(2m + 1)(m = 0, 1, 2 . . .) (5) 𝜎 = 𝜂𝑑𝑐𝑜𝑠 𝜃 λ2𝑚 𝑚 =0,1,2… σb = η f ilmd f ilmcos θ λ(2m)(m = 0, 1, 2 . . .) (6)(6)

wherewhered f𝑑 ilmis the is the thickness thickness and andη f ilm 𝜂is the is refractive the refractive index. index. A ray A of ray light of with light wavelength with wave- λlengthenters λ this enters thin this ﬁlm. thin If film. half If the half wavelength the wavelength is an is odd an odd multiplication, multiplication, there there will will be abe brighta bright constructive constructive interference interference using using (5). (5). On On the the other other hand, hand, if half if half the wavelengththe wavelength is an is evenan even multiplication, multiplication, there there will bewill a darkbe a dark destructive destructive interference interference using using (6). We (6). used We thisused principlethis principle to implement to implement thin ﬁlm thin interventions film interventions and iridescence. and iridescence.

FigureFigure 8. 8.Principle Principle of of thin-ﬁlm thin-film interference. interference.

AccordingAccording to to basic basic theory theory of physicallyof physically based based rendering rendering [26], [26], the physicallythe physically based based ren- deringrendering pipeline pipeline can be can classified be classified into basicinto basic color, color, metallicity, metallicity, roughness, roughness, Fresnel Fresnel reflectance, reflec- refractivetance, refractive index, and index, reflection and reflection of materials. of mate Inrials. addition, In addition, the movement the movement change, change, distortion, dis- opacity,tortion, and opacity, surrounding and surrounding occlusion occlusion can be considered. can be considered. We implemented We implemented the soap the bubble soap to which physically based rendering technology was applied by applying the following procedures. The thin film interference, which is essential for soap bubble rendering, was implemented as a Shader graph using a physically based principles, rather than using Unity’s default shader. The physical parameters used at this time are d f ilm, thickness and η f ilm, refractive. The thickness of thin film was then used as the actual measured parameter value using the author’s previous study, [19]. Additionally, iridescence layer thickness is also defined. Along with the motion vector texture, iridescence changes shape and thickness with wavelengths of color. The default iridescence is visible only at the border, but the shape and thickness of the iridescence layer vary depending on the flow shape of texture. First, we used a Shader graph, a node-based Shader editor of Unity, to implement the soap bubbles as physically based rendering, and we applied each vertex node and parameter to the master node.

4.3. Comparison of Motion Change of Height Map and Motion Vector Texture The change in motion of the ﬂuid was compared with the case of using the height map and the case of using the motion vector texture for the UV coordinate of the texture of the soap bubble, which enabled the sampling of the ﬂow shape, thickness, and color value. As shown in Figure9, there was a clear difference in the light source. When using the height map, only the light source was observed on the surface. The motion vector texture shows Appl. Sci. 2021, 11, x FOR PEER REVIEW 11 of 18

bubble to which physically based rendering technology was applied by applying the following procedures. The thin film interference, which is essential for soap bubble rendering, was implemented as a Shader graph using a physically based principles, rather than using Unity’s default shader. The physical parameters used at this time are 𝑑, thickness and 𝜂, refractive. The thickness of thin film was then used as the actual measured parameter value using the author’s previous study, [19]. Additionally, iridescence layer thickness is also defined. Along with the motion vector texture, iridescence changes shape and thickness with wavelengths of color. The default iridescence is visible only at the border, but the shape and thickness of the iridescence layer vary depending on the flow shape of texture. First, we used a Shader graph, a node-based Shader editor of Unity, to implement the soap bubbles as physically based rendering, and we applied each vertex node and parameter to the master node.

4.3. Comparison of Motion Change of Height Map and Motion Vector Texture The change in motion of the fluid was compared with the case of using the height Appl. Sci. 2021, 11, 3090 map and the case of using the motion vector texture for the UV coordinate of the texture11 of 17 of the soap bubble, which enabled the sampling of the flow shape, thickness, and color value. As shown in Figure 9, there was a clear difference in the light source. When using the height map, only the light source was observed on the surface. The motion vector thetexture color shows response the fromcolor theresponse light source, from the and light the source, flow shape and andthe flow irregular shape velocity and irregular of the fluidvelocity can of be the observed. fluid can It canbe observed. be seen that It can the be height seen map that ofthe the height grayscale map isof notthe suitablegrayscale for is UVnot Coordinatesuitable for andUV Coordinate is controlled and by theis controlled red and green by the values red and of green the motion values vector of the texture. motion Invector addition, texture. the colorIn addition, response the observed color response from the observed light source from can the be light variously source changedcan be vari- by adjustingously changed the smoothness by adjusting intensity the smoothness parameter. intensity In this parameter. way, when In using this way, a motion when vector using texturea motion than vector when texture using only than the when height using map, only the the color height response map, is the displayed color response on the surface, is dis- asplayed well ason the the direction surface, as and well velocity as the values direction in each and pixel,velocity providing values in more each realistic pixel, providing changes inmore the motionrealistic of changes the fluid. in the motion of the fluid.

FigureFigure 9. 9.Comparison Comparison of of motion motion change change of of height height map map on on the the top top row row and and motion motion vector vector texture texture on on the bottom row. the bottom row.

5.5. ExperimentalExperimental ResultsResults TheThe computer computer speciﬁcations specifications used used in thisin this research research were were CPU CPU (AMD (AMD Ryzen5 Ryzen5 3600), 3600), GPU (NVIDAGPU (NVIDA GeForce GeForce RTX 2070 RTX SUPER),2070 SUPER), RAM RAM (16 GB), (16GB), but the but minimum the minimum speciﬁcations specifications for computersfor computers for actual for actual simulation simulation operation operation can be can CPU be (IntelCPU Core(Intel i5-4590/AMDCore i5-4590/AMD FX 8350 FX or8350 higher), or higher), and GPU and (NVIDIA GPU (NVIDIA GeForce GeForce GTX 1060RMD) GTX 1060RMD) In addition, In addition, HTC VIVE HTC Pro VIVE (HMD) Pro for(HMD) virtual for reality virtual and reality Leap and motion Leap controller motion forcontroller interactive for handinteractive motion hand tool motion were used. tool were used. 5.1. Soap Bubble Comparison

Bubble shows take place in various environments and with different light conditions. For comparative analysis, we compared and analyzed soap bubbles in three settings. Figure 10a shows the results created by the proposed study, and soap bubbles in Figure 10b,c have been simulated according to methods [27,28] in Unity. Soap bubbles are basically fluids, so they reflect, refract and allow passage of light. However, the examples in Figure 10b,c are not suitable for reflection or refraction of light, as observed from the surface. In addition, soap bubbles are fluid, and at the same time, the soap foil has an interference pattern due to thin film interference. Figure 10b shows some interference patterns in the border, but it is not realistically expressed. In addition, interference fringes cannot be found in (c). On the other hand, in the cases of soap bubbles (a) to which the physically based rendering technology of this study is applied, it can be confirmed that the interference pattern and the reflected color appear different according to the viewing distance and angle in real time (b) and (c). It is possible to know the change in motion according to the flow of fluid that cannot be observed in (b) and (c), since the existing soap bubbles focus on the shape, generation and bursting for simulation, adapting to the surrounding environment, changing the motion according to the flow of fluid, and changing the color according to the viewing angle, etc. However, (a) could be able to provide acceptable quality soap bubbles by solving these problems and will contribute to the reproduction of bubble show. Appl. Sci. 2021, 11, x FOR PEER REVIEW 12 of 18

5.1. Soap Bubble Comparison Bubble shows take place in various environments and with different light conditions. For comparative analysis, we compared and analyzed soap bubbles in three settings. Fig- ure 10a shows the results created by the proposed study, and soap bubbles in Figure 10b,c have been simulated according to methods [27,28] in Unity. Soap bubbles are basically fluids, so they reflect, refract and allow passage of light. However, the examples in Figure 10b,c are not suitable for reflection or refraction of light, as observed from the surface. In addition, soap bubbles are fluid, and at the same time, the soap foil has an interference pattern due to thin film interference. Figure 10b shows some interference patterns in the border, but it is not realistically expressed. In addition, interference fringes cannot be found in (c). On the other hand, in the cases of soap bubbles (a) to which the physically based rendering technology of this study is applied, it can be confirmed that the interference pattern and the reflected color appear different according to the viewing distance and angle in real time (b) and (c). It is possible to know the change in motion according to the flow of fluid that cannot be observed in (b) and (c), since the existing soap bubbles focus on the shape, generation and bursting for simulation, adapting to the surrounding environment, changing the motion according to Appl. Sci. 2021, 11, 3090 the flow of fluid, and changing the color according to the viewing angle, etc. However, 12(a) of 17 could be able to provide acceptable quality soap bubbles by solving these problems and will contribute to the reproduction of bubble show.

(a) (b) (c)

FigureFigure 10.10. ((aa)) Ours.Ours. ( (b)) Iridescence Iridescence Soap Soap Bubble Bubble [27], [27 ],and and (c) ( cJiggly) Jiggly Bubble Bubble [28]. [28 ].

5.2. Implementation through the Soap Bubble Tool There are various tools for handling soap bubbles to reproduce a bubble show. In this study, tools were largely classiﬁed into “straw1”, “straw2”, “tube”, and “hoop”. Straws were subdivided into a large straw (1.3 cm) and a small straw (0.7 cm). For this, Unity version 2019.3.5f1 was used, and the selection and use of tools was performed with Leap Motion. The menu was composed of an upper menu and a lower menu, and was implemented in a HUD format on the left hand. Figure 11a shows a user playing in a virtual reality environment wearing a VR HMD, and Figure 11b shows the upper menu, which appears when you open the thumb of your left hand, and there are “Select Tool” and “Quit” options, the latter performing the HUD shutdown function. The Select Tool allows the user to move to the lower menu as shown in Figure 11c. In the lower menu, the user can select tools such as hoop, straw1, straw2, and tube from the right, and through this, you can experience the soap bubble show. It also has an initialization function that allows you to select other tools.

5.3. Soap Bubbles with Optimized Modeling Figure 12 shows soap bubbles blowing out of a straw by applying physically based rendering technology. Figure 12 gathers the appearance of soap bubbles created using small straws. Figure 12a is the ﬁrst soap bubble in the process of blowing out. During the blowing, you can see that only the cylindrical object is created and that no soap bubbles are created. After the cylindrical object with a separation point is created, bubbles are created at that location. Figure 12b shows a side view of the handling tools. You can see Appl. Sci. 2021, 11, x FOR PEER REVIEW 13 of 18

5.2. Implementation through the Soap Bubble Tool There are various tools for handling soap bubbles to reproduce a bubble show. In this study, tools were largely classified into “straw1”, “straw2”, “tube”, and “hoop”. Straws were subdivided into a large straw (1.3 cm) and a small straw (0.7 cm). For this, Unity version 2019.3.5f1 was used, and the selection and use of tools was performed with Leap Motion. The menu was composed of an upper menu and a lower menu, and was implemented in a HUD format on the left hand. Figure 11a shows a user playing in a virtual Appl. Sci. 11 2021, , 3090 reality environment wearing a VR HMD, and Figure 11b shows the upper menu, which13 of 17 appears when you open the thumb of your left hand, and there are “Select Tool” and “Quit” options, the latter performing the HUD shutdown function. The Select Tool allows thethe soapuser to bubbles move to spread the lower out menu in several as shown branches in Figure along 11c. the In moving the lower path menu, deﬁned the user above. Figurecan select 12c toolsshows such the as tools hoop, being straw1, handled straw2, from and the tube user’s from point the right, of view and and through soap this, bubbles beingyou can blown experience out. the soap bubble show. It also has an initialization function that allows you to select other tools.

(a) (b) (c)

Appl. Sci. 2021, 11, x FOR PEER REVIEWFigureFigure 11. VRVR HMD HMD (a ()a upper) upper menu menu (b) (andb) andtools tools menu menu (c) for ( ccreating) for creating soap bubbles soap bubbles in virtual14 in of virtual18 reality. Appl. Sci. 2021, 11, x FOR PEER REVIEWreality. 14 of 18

5.3. Soap Bubbles with Optimized Modeling Figure 12 shows soap bubbles blowing out of a straw by applying physically based rendering technology. Figure 12 gathers the appearance of soap bubbles created using small straws. Figure 12a is the first soap bubble in the process of blowing out. During the blowing, you can see that only the cylindrical object is created and that no soap bubbles are created. After the cylindrical object with a separation point is created, bubbles are created at that location. Figure 12b shows a side view of the handling tools. You can see the soap bubbles spread out in several branches along the moving path defined above. Figure 12c shows the tools being handled from the user’s point of view and soap bubbles being blown out.

(a) (b) (c)

Figure 12. Soap( abubbles) created through a straw:(b The) first bubble in the process of(c being) blown Figure 12. Soap bubbles created through a straw: The ﬁrst bubble in the process of being blown out out (a), a side view of the handling tools (b), and the user’s point of view as bubbles are blown out (Figurea), a side 12. view Soap ofbubbles the handling created toolsthrough (b), a and straw: the The user’s first point bubble of viewin the as process bubbles of arebeing blown blown out of the ofout the (a ),straw a side (c ).view of the handling tools (b), and the user’s point of view as bubbles are blown out straw (c). of the straw (c). Figure 13a shows the appearance of soap bubbles beginning to come out of a large Figure 13a shows the appearance of soap bubbles beginning to come out of a large tube, tube, Figurewhich 13ahave shows a greater the appearance diameter than of soap stra ws.bubbles It can beginning be seen thatto come the outsoap of bubbles a large which have a greater diameter than straws. It can be seen that the soap bubbles blowing blowingtube, which out fromhave thea greater ends of diameter the straw than are larger straws. than It canthe straw,be seen and that that the they soap sag bubbles down- out from the ends of the straw are larger than the straw, and that they sag downwards wardsblowing under out from the influence the ends ofof gravity.the straw Figure are larger 13b shows than the the straw, bubble’s and appearance that they sag immedi- down- underatelywards after theunder inﬂuencethe the soap influence bubble of gravity. of separates gravity. Figure Figurefrom 13 bth 13b showse tube shows and the the bubble’sFigure bubble’s 13c appearance appearanceshows soap immediately immedi-bubbles aftercomingately theafter out soap the of bubblethesoap tube bubble separates continuously. separates from from the tube the andtube Figureand Figure 13c shows 13c shows soap soap bubbles bubbles coming outcoming of the out tube of the continuously. tube continuously.

(a) (b) (c) Figure 13. Soap( abubbles) generated through the tube:(b) A bubble beginning to leave( thec) tube (a), a bubble after it has left the tube (b), and bubbles being continuously blown out of the tube (c). FigureFigure 13. SoapSoap bubbles bubbles generated generated through through the thetube: tube: A bubble A bubble beginning beginning to leave to the leave tube the (a tube), a (a), a bubble after it has left the tube (b), and bubbles being continuously blown out of the tube (c). bubbleFigure after it14 has shows left the the tube appearance (b), and bubblesof soap beingbubbles continuously being blown blown out outof the of the hoop tube by (c ap-). plyingFigure physically 14 shows based the rendering appearance technology. of soap bu Thebbles hoop being recognizes blown out the of hand the hoopthrough by ap-lip motion,plying physically and when based the user rendering holds the technology. hoop in theThe desired hoop recognizes direction andthe hand moves, through it recog- lip nizesmotion, the and direction when andthe userbubbles holds come the out.hoop The in initialthe desired soap bubbledirection model and ofmoves, using it hoop recog- is madenizes theof 3D direction MAX and and then bubbles load itcome into aout. program The initial to reproduce soap bubble the processmodel ofof usingdeformation. hoop is Itmade is simulated of 3D MAX through and then three load stages it into of adeform programation. to reproduce The first is the the process moment of soapdeformation. bubbles areIt is produced, simulated thethrough same threeas the stages round of size deform of theation. hoop. The At first this istime, the momentsoap bubbles soap arebubbles gen- eratedare produced, in a natural the samedirection as the by round calculating size of th thee absolute hoop. At values this time, that thesoap user bubbles moves are on gen- the X,erated Y, and in aZ natural axes. Second, direction the by maximum calculating length the absolute step is valuesthe moment that the when user soapmoves bubbles on the haveX, Y, theand longest Z axes. form. Second, It is thelong maximum horizontally length and step is influenced is the moment by gravity, when sosoap the bubblesmiddle parthave is the relatively longest downward. form. It is long In the horizontally circular phase, and itis meansinfluenced that theby gravity,horizontally so the stretched middle bubblespart is relatively gradually downward. return to theIn the circular circular mode phase,l. The it meansmovement that theof soaphorizontally bubbles stretched used air resistancebubbles gradually acceleration return and to falling the circular acceleration mode tol. Thereflect movement reality, and of thesoap movement bubbles used of soap air bubblesresistance increased acceleration and decrand fallingeased by acceleration using morphing. to reflect reality, and the movement of soap bubbles increased and decreased by using morphing. Appl. Sci. 2021, 11, 3090 14 of 17

Figure 14 shows the appearance of soap bubbles being blown out of the hoop by applying physically based rendering technology. The hoop recognizes the hand through lip motion, and when the user holds the hoop in the desired direction and moves, it recognizes the direction and bubbles come out. The initial soap bubble model of using hoop is made of 3D MAX and then load it into a program to reproduce the process of deformation. It is simulated through three stages of deformation. The first is the moment soap bubbles are produced, the same as the round size of the hoop. At this time, soap bubbles are generated in a natural direction by calculating the absolute values that the user moves on the X, Y, and Z axes. Second, the maximum length step is the moment when soap bubbles have the longest form. It is long horizontally and is influenced by gravity, so the middle part is relatively downward. In the circular phase, it means that the horizontally stretched bubbles gradually return to the circular model. The movement of soap bubbles used air resistance Appl. Sci. 2021, 11, x FOR PEER REVIEWacceleration and falling acceleration to reflect reality, and the movement of soap bubbles15 of 18 increased and decreased by using morphing.

(a) (b) (c)

FigureFigure 14. 14.Soap Soap bubbles bubbles generated generated through through the the hoop: hoop: AA bubblebubble isis beingbeing blownblown outout ofof thethe hoophoop ((aa),), the bubble is then at its longest just after separation due to the action of gravity (b), and the bubble the bubble is then at its longest just after separation due to the action of gravity (b), and the bubble then becomes more circular over time (c). then becomes more circular over time (c).

FigureFigure 14 14aa shows shows the the first first recognition recognition of of the the direction direction of of the the hoop hoop and and soap soap bubbles bubbles areare blowing blowing out. out. Due Due to to the the nature nature of theof the hoop, hoop, it can it can be seenbe seen that that it comes it comes out longerout longer in the in horizontalthe horizontal plane plane rather rather than circular.than circular. Figure Figure 14b shows 14b shows the appearance the appearance immediately immediately after theafter soap the bubbles soap bubbles come out come and out separate and separate from the from hoop. the The hoop. soap The bubble soap is bubble in its longest is in its state,longest and state, the middleand the part middle is being part is acted being on acted by gravity—you on by gravity—you can see can that see it is that relatively it is rel- droopingatively drooping down. Figuredown. 14Figurec shows 14c shows that the that soap the bubblessoap bubbles are separated are separated and graduallyand gradu- morphally morph into a into spherical a spherical shape shape over time.over time. UsingUsing Leap Leap Motion, Motion, a controllera controller that that recognizes recognizes hand hand gestures, gestures, it supports it supports interactions interac- totions reproduce to reproduce a soap bubblea soap bubble show in show a VR in environment. a VR environment. The hand The motion hand wasmotion performed was per- throughformed handthrough motion hand recognized motion recognized through Leap through Motion. Leap The Motion. hand The movement hand movement for this was for performedthis was performed through the through gesture the grammar gesture grammar of both hands of both as hands follows. as Figurefollows. 15 Figurea is a clap,15a is performeda clap, performed through through the gesture the ofgesture clapping of cla withpping both with hands, both andhands, bubbles and bubbles are created are cre- as theated left as and the rightleft and hands right collide. hands Figure collide. 15 Figureb shows 15b a grab,shows performed a grab, performed through thethrough gesture the ofgesture picking of up picking an object up withan object the leftwith hand the andleft catchinghand and the catching soap bubble the soap while bubble colliding while withcolliding the bottom with the of thebottom left handof the and left thehand soap and bubble. the soap Figure bubble. 15 cFigure shows 15c a fanning shows a motion, fanning performedmotion, performed through a through gesture ofa gest shakingure of the shaking left hand the from left hand side to from side, side and to bubbles side, and within bub- ables certain within distance a certain that collidedistance with that the collide left hand with will the then left fly. hand Figure will 15 thend is bouncing,fly. Figure which 15d is isbouncing, performed which through is performed a gesture through of shaking a gesture the right of shaking palm up the and right down, palm and up whenand down, the soapand bubbleswhen the collide soap bubbles with the collide right palm,with the it bounces right palm, by receivingit bounces force by receiving from the force physical from characteristics.the physical characteristics. Figure 15e is piercing, Figure 15e performed is piercing, through performed a pricking thro gestureugh a pricking with the gesture index fingerwith the of the index right finger hand, of and the whenright hand, the index and fingerwhen andthe index soap bubblesfinger and collide, soap the bubbles bubbles col- burstlide, andthe bubbles disappear. burst In and this disappear. way, it is possible In this way, to interact it is possible in a VR to environment interact in a throughVR envi- handronment motions. through hand motions.

Appl. Sci. 2021, 11, 3090 15 of 17 Appl. Sci. 2021, 11, x FOR PEER REVIEW 16 of 18

(a)

(b)

(c)

(d)

(e)

FigureFigure 15. 15.Soap Soap bubblesbubbles generated th throughrough hand hand motions. motions. (a) ( aClap) Clap (Both (Both hands hands gesture) gesture) (b) Grab (b) Grab (left hand gesture) (c) Fanning (left hand gesture) (d) Bouncing (right hand gesture) (e) Piercing (left hand gesture) (c) Fanning (left hand gesture) (d) Bouncing (right hand gesture) (e) Piercing (right (right hand gesture). hand gesture). Appl. Sci. 2021, 11, 3090 16 of 17

6. Conclusions For various VR content, including bubble shows, the realistic expression of bubbles is very important. In this study, a realistic soap bubble reproduction method was proposed by applying a physically based rendering technology for bubble art that can produce an immersive feeling. First, the characteristics of the soap bubble fluid were analyzed by considering the physically based rendering pipeline. Second, we applied the algorithm available in Shader Graph, a node-based shader editor of Unity. Through this, various expressions were possible through the adjustment of parameters. By extracting the coordinates of the measured path, the path through which the soap bubble moved was created, and the surface of the bubble was implemented by using the difference between the internal and external pressure, assuming the surface tension as a constant. In addition, the produced soap bubble was integrated with the model in which the soap bubble came out by applying the diameter and number measured through the experiment, and a shader where the soap film surface flowed over time was applied. The integrated soap bubble was called with a predetermined size according to the model and moved along the stored path. When it was called out, the various shapes of soap bubbles could be observed moving toward a sphere. In addition, the soap film flowed and its surface gradually became transparent, resulting in a more realistic soap bubble simulation being produced. To provide better quality soap bubbles in the future, we intend to develop more advanced physically based rendering pipeline and its parameters. First, it needs to convey physically measured parameters of soap bubbles according to the degree of surface activity. Bubble art is a necessary element because it mixes various materials to express artistic and unusual soap bubbles. Second, it is necessary to optimize the fluid motion change according to the tool or environment. Most of the fluids, including soap bubbles, were either ignored or expressed uniformly. In this study, a motion vector texture was used to solve this problem, but the change in motion varies greatly depending on the tool or environment. However, there are still limitations to controlling tools and environments. Nevertheless, as optimized modeling becomes possible, this study contributes to soap bubbles being applied with physically based rendering technology to various VR content.

Author Contributions: Conceptualization, S.Y. and S.C.; methodology, S.Y., C.L. and S.C.; software, S.Y. and C.L.; validation, S.Y. and S.C.; formal analysis, C.L.; investigation, S.Y. and C.L.; resources, S.Y.; data curation, C.L.; writing—original draft preparation, S.Y., C.L. and S.C.; writing—review and editing, S.Y. and S.C.; visualization, S.Y.; supervision, S.C.; project administration, S.C.; funding acquisition, S.C. All authors have read and agreed to the published version of the manuscript. Funding: This research was supported by Basic Science Research Program through the National Re- search Foundation of Korea (NRF) funded by the Ministry of Education (No. 2018R1D1A1B0704256613). Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: Not applicable. Conﬂicts of Interest: The authors declare no conﬂict of interest.

References 1. Eom, I. A Study on Content Classification for Developing Virtual Reality-based Attraction Contents. J. Korea Contents Assoc. 2019, 19, 499–506. 2. Salkin, L.; Schmit, A.; Panizza, P.; Courbin, L. Generating soap bubbles by blowing on soap films. Phys. Rev. Lett. 2016, 116, 077801. [CrossRef][PubMed] 3. Prosperetti, A. Bubbles. Phys. Fluids 2004, 16, 1852–1865. [CrossRef] 4. Da, F.; Batty, C.; Wojtan, C.; Grinspun, E. Double bubbles sans toil and trouble: Discrete circulation-preserving vortex sheets for soap films and foams. ACM Trans. Graph. 2015, 34, 1–9. [CrossRef] 5. Kim, N.; Oh, S.; Park, K. Giant soap bubble creation model. Comput. Animat. Virtual Worlds 2015, 26, 445–455. [CrossRef] 6. Morris, N. Capturing the Reflectance Model of Soap Bubbles. Univ. Tor. 2003.[CrossRef] Appl. Sci. 2021, 11, 3090 17 of 17

7. Afanasyev, Y.D.; Andrews, G.T.; Deacon, C.G. Measuring soap bubble thickness with color matching. Am. J. Phys. 2011, 79, 1079–1082. [CrossRef] 8. Sane, A.; Mandre, S.; Kim, I. Surface tension of flowing soap films. J. Fluid Mech. 2018, 841. [CrossRef] 9. Iwasaki, K.; Matsuzawa, K.; Nishita, T. Real-time rendering of soap bubbles taking into account light interference. Proc. Comput. Graph. Int. IEEE 2004, 2004, 344–348. 10. Botta, O.D.; Magos, I.; Balan, C. Experimental study on the formation and break-up of fluid bubbles. Incas Bull. 2020, 12, 27–34. [CrossRef] 11. Davidson, J.; Ryu, S. High-speed visualization of soap bubble blowing and image-processing-based analysis of pinch-off dynamics. J. Vis. 2017, 20, 53–61. [CrossRef] 12. Biria, A.; Fried, E. Theoretical and experimental study of the stability of a soap film spanning a flexible loop. Int. J. Eng. Sci. 2015, 94, 86–102. [CrossRef] 13. Pfeiffer, P.; Ohl, C.D. Spreading of soap bubbles on dry and wet surfaces. Sci. Rep. 2020, 10, 1–9. [CrossRef][PubMed] 14. Glassner, A. Soap bubbles: Part 2. IEEE Ann. Hist. Comput. 2020, 20, 99–109. 15. Ishida, S.; Synak, P.; Narita, F.; Hachisuka, T.; Wojtan, C. A model for soap film dynamics with evolving thickness. ACM Trans. Graph. 2020, 39, 31-1. [CrossRef] 16. Vorobieff, P.; Rivera, M.; Ecke, R.E. Soap film flows: Statistics of two-dimensional turbulence. Phys. Fluids 1999, 11, 2167–2177. [CrossRef] 17. Huang, W.; Iseringhausen, J.; Kneiphof, T.; Qu, Z.; Jiang, C.; Hullin, M.B. Chemomechanical simulation of soap film flow on spherical bubbles. Acm Trans. Graph. 2020, 39, 41-1. [CrossRef] 18. Halliday, D.; Resnick, R. Fundamentals of Physics; John Wiley Sons: Hoboken, NJ, USA, 1981. 19. Gil, S.; Seok, Y.; Park, K.; Yoo, J.; Chin, S. Soap film flow and thickness for soap bubble rendering. In Proceedings of the 25th ACM Symposium on Virtual Reality Software and Technology, Parramatta, NSW, Australia, 12–15 November 2019; pp. 1–2. 20. Han, S.; Yoo, S.; Chin, S. Physics-based height map optimization conveying real-measured flow speed for virtual soap bubble rendering. Int. J. Adv. Cult. Technol. 2020, 8, 284–290. 21. Hassan, M. Proposed Workflow for UV Mapping and Texture Painting. 2016. Available online: https://www.diva-portal.org/ smash/record.jsf?pid=diva2%3A945618&dswid=-7551 (accessed on 25 April 2020). 22. Chai, T.; Draxler, R.R. Root mean square error (RMSE) or mean absolute error (MAE)?–Arguments against avoiding RMSE in the literature. Geosci. Model Dev. 2014, 7, 1247–1250. [CrossRef] 23. Hirayama, H.; Yamaji, Y.; Kaneda, K.; Yamashita, H.; Monden, Y. Rendering iridescent colors appearing on natural objects. In Proceedings of the Eighth Pacific Conference on Computer Graphics and Applications IEEE, Hong Kong, China, 5 October 2000; pp. 15–433. 24. Smith, M. Unity 2018 Cookbook: Over 160 Recipes to Take Your 2D and 3D Game Development to the Next Level; Packt Publishing Ltd.: Birmingham, UK, 2018. 25. Sadeghi, I.; Jensen, H.W. A physically based anisotropic iridescence model for rendering morpho butterflies photo-realistically. In Project Report for Advanced Appearance Modeling Course CSE272 Fall; University of California San Diego: San Diego, CA, USA, 2007. 26. Russell, J. Basic Theory of Physically-Based Rendering. 2017. Available online: https://marmoset.co/posts/basic-theory-of- physically-based-rendering/ (accessed on 12 March 2020). 27. Smkplus. Available online: https://github.com/smkplus/Iridescence.git (accessed on 10 September 2020). 28. Moonflower. Available online: https://assetstore.unity.com/packages/vfx/particles/environment/jiggly-bubble-free-61236 (accessed on 20 September 2020).