3D Model Streaming Based on JPEG 2000 Nein-Hsien Lin, Ting-Hao Huang, and Bing-Yu Chen, Member, IEEE Abstract — For PC and even mobile device, video and as what has been done in the video and image streaming. image streaming technologies, such as H.264 and Therefore, in this paper, we propose a mesh streaming JPEG/JPEG 2000, are already mature. However, the method by utilizing the benefits of JPEG 2000 (J2K) [17]. streaming technology for 3D model or so-called mesh data is This method takes the advantages of geometry image [5] still far from practical use. Therefore, in this paper, we which can convert the mesh data of a 3D model into an image. propose a mesh streaming method based on JPEG 2000 This could reduce the problem of 3D mesh streaming and standard and integrate it into an existed multimedia transfer it to a 2D image streaming problem. There are also streaming server, so that our mesh streaming method can many 2D image compression methods that can be used to directly benefit from current image and video streaming further downsize the file of the geometry image. This paper technologies. In this method, the mesh data of a 3D model is made use of the JPEG 2000 compression due to some of its first converted into a JPEG 2000 image, and then based on good characteristics, such as ROI (Region Of Interest), the JPEG 2000 streaming technique, the mesh data can then progressive compression, multiple components, etc. Moreover, be transmitted over the Internet as a mesh streaming. Furthermore, we also extend this mesh streaming method since this method is based on JPEG 2000, which is a famous for deforming meshes as the extension from a JPEG 2000 standard, our system is much suitable to be integrated into any image to a Motion JPEG 2000 video, so that our mesh existed client-server or pear-to-pear multimedia streaming streaming method is not only for transmitting a static 3D system. model but also a 3D animation model. To increase the Based on our mesh streaming method, during the usability of our method, the mesh stream can also be downloading process, the user can first obtain an approximate inserted into a X3D scene as an extension node of X3D. shape and then the 3D model will become clearer when more Moreover, since this method is based on the JPEG 2000 data is received. Besides the progressive transmission, the standard, our system is much suitable to be integrated into view-dependent issue is also taken into consideration. Hence, any existed client-server or pear-to-pear multimedia the most significant part of the 3D model or the part faced to streaming system1. the user will be refined earlier. Furthermore, since the 3D animation models or so-called deforming meshes are used Index Terms — Mesh Streaming, JPEG 2000, Geometry more frequently than static 3D models, we also extent our Image, X3D. mesh streaming method for transmitting a 3D animation model as a deforming meshes streaming by converting a 3D I. INTRODUCTION animation model into a Motion JPEG 2000 video. To increase Recently, 3D graphics over the Internet has attracted a lot the usability of our mesh streaming method, the 3D model or of attention, such as web-based virtual shopping malls, on-line 3D animation model represented by a JPEG 2000 image or a 3D games, etc. For supporting these applications, the demand Motion JPEG 2000 video can be inserted into a X3D scene as of transmitting 3D models increased significantly. Being able an extension node of X3D. to view or play a 3D model or a 3D scene composed of many This paper is organized as following. In Section II, the sophisticated 3D models over the Internet is one of the goals related work is introduced. In Sections III and IV, we describe of VRML (Virtual Reality Modeling Language) [16] and X3D the details of our mesh streaming method and the network (eXtensible 3D) [20]. However, due to the increase of the architecture of our mesh streaming system. Sections V and VI model complexity and file size, even with the increase of illustrate the results and a short discussion on the future work network bandwidth, to download the 3D models would still of this method. take a lot of time. To reduce the waiting time for downloading the 3D models, mesh streaming mechanism must be available II. RELATED WORK 1 This work is partially supported by the National Science Council of In this session, we first introduce some background related Taiwan under the numbers: NSC92-2218-E-002-056, NSC93-2622-E-002-033, to the main themes of this paper including 3D streaming, NSC94-2622-E-002-024, and NSC95-2221-E-002-273. geometry images, and image compression. Nein-Hsien Lin is with the National Taiwan University (e-mail: [email protected]). A. Mesh Streaming and Compression Ting-Hao Huang is with the National Taiwan University (e-mail: richardg@ cmlab.csie.ntu.edu.tw). In general, the main goal of mesh streaming is to provide Bing-Yu Chen is with the National Taiwan University (e-mail: 3D contents in real-time for users over the network links, such [email protected]). that the interactivity and visual qualities of the contents may coordinates (,)uvii of the vertex i . Hence, the number of match as closely as when they were stored locally. The vertices N of the reconstructed 3D model will be the same as resource bottleneck is often assumed to be the bandwidth and the resolution of the geometry image wh× , i.e., Nwh= × . not rendering or processing power [14]. To achieve this, To reconstruct the 3D model from a geometry image, we simplification and progressive transmission are two dominant first convert the R, G, B values (,R GB ,) of each pixel i of strategies [12]. Existing mesh streaming techniques may be iii the geometry image to be vertex positions (,x yz ,) of vertex categorized into four main types [3], and in this paper the iii main point we focus is about the object streaming using i . Then, we connect the vertex i to its horizontal and vertical geometry images. neighbors to be several quadrangles, and then we choose the Hoppe introduced the concept of progressive meshes (PM) shorter distance between the two diagonals of each quadrangle [6], which store an arbitrary triangular mesh M as an and connect the opposite vertices. Hence, the reconstructed 3D model can be represented as a triangular mesh. Finally, the appearance-preserving but much coarser base mesh M0 and a vertex attributes, such as normal vectors (,,)NNN or number of refinement pieces constructed by using iterative xyziii edge collapse operations. Then, the 3D model can be texture coordinates (,)uvii, can also be assigned to the vertex represented as M0 , M1 , M2 , ..., MMn = , where M1 , M2 , ..., i . and Mn are the refined multi-resolution meshes by splitting To extend the geometry images for 3D animation model or so-called deforming meshes, Briceño et al. present geometry one vertex in the previous meshes M0 , M1 , ..., Mn−1 . A user videos [1] to provide a new data structure to encode in the client may view or interact immediately with the 3D deforming meshes. Their data structure provides a way to treat model once the base mesh M is downloaded. Streaming 0 deforming meshes as a video sequence and is well suitable for additional parts incrementally refines the mesh and restores network streaming. They also offer the possibility to apply MM= the original mesh n exactly. Geometrical meshes thus and adapt existing mature video processing and compression can be streamed from servers to clients, making interactions techniques, such as MPEG. with 3D data possible without a complete download [2]. C. JPEG 2000 and JPIP A different concept introduced by Isenburg and Lindstrom is streaming meshes (SM) [9]. They reorganize the triangle mesh, so that highly spatial related triangles are packed with each other in the file. While rendering a triangle, other triangles close to it will be found locally in the file, thus avoids most file seek system calls. So we can see this mesh clearly from some parts to entire. The main goal of this (a) scheme is designed for memory out-of-core situation. The third concept is to compress the 3D model [15]. Originally this technique is designed for the limited bandwidth between CPU and GPU, but it can also be used for network transmission. There are two categories of mesh compression: geometry compression and connectivity compression, where (b) the geometry compression dominates the final file size. To compress a 3D model, the vertex data is first quantized, and then some predictions based on observations are made. These predictions can provide clues for entropy/arithmetic coding. In this paper, we use the properties of JPEG 2000 to (c) combine the concepts of PM and SM, so that a remote user can view the 3D model not only progressively but also clearly from user’s viewpoint. B. Parameterization and Geometry Image Surface parameterization is to find a mapping function F:RR32↔ which maps 3D coordinates (,x yz ,) to 2D (d) Fig. 1. The four types of progression provided by JPEG 2000: (a) coordinates (,)s t , i.e., F(x ,yz , )= ( st , ) . Geometry images [5] progression by quality; (b) progression by locality; (c) progression by provided by Gu et al. is one of the surface parameterization component; (d) progression by resolution. methods. In this method, given a 3D model, we can use a 2D The JPEG 2000 standard supports lossy and lossless image to represent it, where the R, G, B values (,R GB ,) of iii compression of single-component (e.g., grayscale) and multi- each pixel i of the image are used to represent the 3D component (e.g., color) imagery.