Data-driven Flower Petal Modeling with Botany Priors Chenxi Zhang Mao Ye Bo Fu Ruigang Yang Center for Visualization & Virtual Environments University of Kentucky Abstract make this modeling problem tractable, we develop a unique pipeline that incorporate domain-specific knowledge. More In this paper we focus on the 3D modeling of flower, in specifically, the shape space of petals (the most dominant particular the petals. The complex structure, severe occlu- components of a flower) can be learned from individually sions, and wide variations make the reconstruction of their scanned petals and their relative spatial layout can be known 3D models a challenging task. Therefore, even though the a priori from botany study. flower is the most distinctive part of a plant, there has been Our approach focuses on the parametric modeling of little modeling study devoted to it. We overcome these chal- flower petals. It starts with the data capture process of a lenges by combining data driven modeling techniques with single flower, for which we use a structured light scanning domain knowledge from botany. Taking a 3D point cloud system consisting of one camera and one projector to cap- of an input flower scanned from a single view, our method ture its shape in 3D. Once we capture the geometric details starts with a level-set based segmentation of each individ- of a flower as a point cloud, our proposed method segments ual petal, using both appearance and 3D information. Each it into different components (petals) based on both 2D ap- segmented petal is then fitted with a scale-invariant mor- pearance and 3D depth information. Each segmented petal phable petal shape model, which is constructed from indi- is then fitted using a scale-invariant morphable petal shape vidually scanned exemplar petals. Novel constraints based model built from individually scanned single petal samples. on botany studies, such as the number and spatial layout of In our setup, flowers are captured from a single view (top petals, are incorporated into the fitting process for realisti- view). The tightly overlaying flower petals make multi- cally reconstructing occluded regions and maintaining cor- view capture less effective. To handle the occlusions as well rect 3D spatial relations. Finally, the reconstructed petal as to maintain the semantic layout of flowers, a set of novel shape is texture mapped using the registered color images, constraints derived from botany studies and segmentation with occluded regions filled in by content from visible ones. information are incorporated into the petal fitting process. Experiments show that our approach can obtain realistic Finally, the reconstructed flower model is texture mapped modeling of flowers even with severe occlusions and large using the captured color images, and occluded regions are shape/size variations. filled in with texture from other complete components. An overview of our approach is shown in Figure 1. To our knowledge, our system is the first to focus on 1. Introduction flower modeling, petals in particular, from 3D point cloud. Plants modeling is one of the most difficult tasks in com- The key contributions of our work can be summarized as: 1) puter vision and graphics community because of their com- a novel petal fitting algorithm that is robust to significant oc- plex geometry and appearance. Flower, as the most distinc- clusions; 2) a robust scheme for flower petal segmentation, tive part of a plant, has fine structures and wide variations, by extending a two-region level-set formulation to multiple which makes reconstructing their 3D models a challenging regions; 3) a scale-invariant morphable petal shape model task. Existing 3D modeling techniques for plants and veg- which can handle wider shape variations within a species, etation are usually designed for large scale structures, such or even across species. as trees, foliage, or based on pure synthesis given some pre- It should be emphasized that our reconstruction pipeline defined rules and templates. generates a parametric model, which is particularly suited The biggest challenge for flower modeling is occlusion. for measurement, editing, and animation. For example, one The tight formation of flower petals make segmentation could easily apply a geometric morphing between two mod- and 3D reconstruction a very challenging task. In order to els, or make global changes to the shapes by varying shape 1 Figure 1. From left to right: 1) Petal database for Lily species; (2) Input image; (3) Petal segmentation; (4) Scanned 3D data; (5) Recon- structed 3D model. parameters. While these are not explored in the context of are from Quan [20] and Bradley [2]. Quan et al. use simi- this paper, we believe our approach will enable more re- lar modeling procedures to ours, composed of an interactive search in the modeling and animation of an intrinsic class leaf segmentation and template based model fitting. Their of objects, flowers, with applications in botany, entertain- approach requires multi-view data in which the entire plant ment, and visual simulations. is captured, while in our case we only use data from a sin- gle view because multi-view data do not provide significant 2. Related Work more converges due to the tight formation of flower petals. Both [20] and [2] use an exemplar leaf mesh to fit to the Due to their importance in the real world, there are many dense point clouds non-rigidly. [2] further learns a statistical approaches for modeling plants. They can be roughly di- model for continuing fitting other leaves, as well as for leaf vided into two categories: rule-based modeling and data- synthesis when occlusions are too big. We instead require driven modeling. Rule-based methods use compact rules a shape database for flower petals to handle the significant and grammars for building models of plants. As a prime occlusions in flowers. work, a series of approaches based on the idea of L-system After a survey of existing plant modeling techniques, we were developed [9, 16, 17, 18, 19]. The modeling of plant note that flowers, despite being the most significant focus organs, such as leaves and petals, is a much less studied of study for identification, are the least frequently studied, problem. Fowler and colleagues [4] developed a collision- probably due to its complex structure and significant self- based model for the spiral phyllotaxis effect, where plant or- occlusions. Our method uses both a data-driven approach gans are arranged in spiral patterns. Mundermann and col- and knowledge in botany to handle these challenges. laborators [12] used leaf silhouettes to estimate leaf skele- ton and further build leaf shape models. Fuhrer and col- 3. Flower Petal Modeling leagues [5] studied how to model and render small hairs on plants. Reunions and collaborators [21] developed proce- To capture the geometric details, we choose to use struc- dural algorithms to model a number of leaf venation pat- tured light scanner to acquire high quality 3D data of flow- terns. A related work in flower modeling is an interactive ers. Our method starts from scanning individual flower system by Ijiri and collaborators [6]. It has a graphical user petals with variations but from the same species, and build- interface for users to sketch flower models based on botani- ing up a morphable model [1] for petal shape of a certain cal constraints. However those work summarized above are species. A level-set based active contour model is used for purely rule-based, for which the realism and accuracy de- accurately segmenting the 2D image and 3D scanned points pend on the understanding of flower development and the of a whole flower into different components(petals). Both effort of the modelers. 2D appearance and 3D depth information are used to guide the segmentation. Each segmented petal point clouds is sub- Recently with the proliferation of digital cameras and 3D sequently fitted using the morphable model. We propose a scanning devices, there have been a number of data-driven joint multiple petal fitting algorithm using prior knowledge approaches developed specific for tree, small plants, or fo- from Botany about flower spatial layout. Finally, the regis- liage [20, 22, 14, 10, 11, 2]. Typically major tree branches tered color image is used to generate texture maps for the are detected or interactively traced from 3D point clouds or 3D model. We will illustrate each part in details in next images. Leaves are synthesized based on separate scanning, several sections. some heuristics, or mesh-fitting, so that the final model is visually similar to the input data. Approaches in this cat- 3.1. Scale-invariant Morphable Model egory aim to faithfully reconstruct the 3D model of plants based on the input. They usually focus on plants with a large We choose to use a learned morphable shape model to re- number of leaves and the general structure of the whole construct flower petals because of the parametric nature of plant. From algorithm perspective, the most related works the model representation. The benefit of using morphable 3.2. Flower Petal Segmentation There has been some work on segmenting whole flow- ers from a scene [15], but few has been done on segment- ing each individual component (petal) of a flower. The main challenges are from the high appearance similarity and noticeable self-occlusions, which makes the segmentation very challenging. Therefore, we manually specify a cen- Figure 2. Petal database for Pansies(60 exemplars). tral position on each petal as an initialization to guide the segmentation. model is that the optimization affects the entire petal, as We apply distance regularized level set evolution [8] for- opposed to per-vertex based deformation method, therefore mulation to an active contour model [7] to solve the petal can robustly handle occlusions.