Progressive imagery with scalable vector graphics
Georg Fuchsa, Heidrun Schumanna, and Ren´eRosenbaumb aUniversity of Rostock, Institute for Computer Science, 18051 Rostock, Germany; bUC Davis, Institute of Data Analysis & Visualization, Davis, CA 95616 U.S.A.
ABSTRACT Vector graphics can be scaled without loss of quality, making them suitable for mobile image communication where a given graphics must be typically represented in high quality for a wide range of screen resolutions. One problem is that file size increases rapidly as content becomes more detailed, which can reduce response times and efficiency in mobile settings. Analog issues for large raster imagery have been overcome using progressive refinement schemes. Similar ideas have already been applied to vector graphics, but an implementation that is compliant to a major and widely adopted standard is still missing. In this publication we show how to provide progressive refinement schemes based on the extendable Scalable Vector Graphics (SVG) standard. We propose two strategies: decomposition of the original SVG and incremental transmission using (1) several linked files and (2) element-wise streaming of a single file. The publication discusses how both strategies are employed in mobile image communication scenarios where the user can interactively define RoIs for prioritized image communication, and reports initial results we obtained from a prototypically implemented client/server setup. Keywords: Progression, Progressive refinement, Scalable Vector Graphics, SVG, Mobile image communication
1. INTRODUCTION Vector graphics use graphic primitives such as points, lines, curves, and polygons to represent image contents. As those primitives are defined by means of geometric coordinates that are independent of actual pixel resolutions, vector graphics can be scaled without loss of quality. This presents a strong advantage compared to raster imagery and makes vector graphics very suitable for mobile image communication scenarios where a given graphics must typically be represented in high quality for a broad range of viewing devices. One problem with vector graphics is that file sizes increase rapidly as content becomes more detailed, due to the fact that more primitives are required. This can degrade response times and efficiency in mobile settings due to increasing data complexity and transmission requirements. Similar challenges arise for raster images and have been addressed using progressive refinement schemes. Progressive refinement facilitates low-fidelity previews, minimizes redundancy and bandwidth requirements during transfer, and also affords semantic aspects like pre- defined or interactive Regions-of-Interest (RoI) and guided tours through the data.1 Although the application of progression to vector graphics has already been proposed in cartography, taking advantage of the approach using widely accepted standards is still an open research question. In this paper we show how to provide progressive refinement schemes based on the extendable XML-based Scalable Vector Graphics (SVG) standard format.2 The proposed methodology is compliant to the standard. We introduce two methods that vary in their advantages and properties and support a broad range of functionality that is beneficial for interactive and demand-driven image communication, as Regions-of-Interest and Levels-of- Detail. We illustrate and justify the introduced technology with results we obtained from first experiments and conclude that the introduced technology bridges the gap between flexible and efficient raster and compliant high quality vector image communication. We will review work related to progressive raster and vector imagery in the Section 2 in order to lay the foundation for our methods in Section 3. Details to the implementation and initial findings are presented in Section 4. Conclusions and objectives for future work close our contribution in Section 5. Further author information (contact either author): G.F.: E-mail: [email protected], Telephone: +49 381 498 7484 R.R.: E-mail: [email protected], Telephone: ++1 530 754 9470 Figure 1. Progressive refinement (progression) provides first impressions of the content and may also be used to reduce con- sumed system resources. Although well-researched for raster imagery (top) applying progression to vector data (bottom) is still in its infancy.
2. RELATED WORK Raster image communication has always been the main application of progressive refinement (progression). First schemes3 mostly developed to overcome bandwidth constraints already allowed for content previews, but required redundant transfer and handling of image contents. This has later been overcome by the introduction of scalable image compression and transmission schemes. The basic idea here is to organize the encoded data-stream in such a way that the decoding of a truncated stream leads to a restored image with less detail.4 Thus, content previews can be presented during a still running transmission. As all encodings of preview images are an integral part of the data stream created for the final image, a non-redundant data transmission is inherent. To combine the requirements of this procedure with a high compression performance, sophisticated codecs such as SPIHT5 or JPEG20006 have been developed. With progression as its foundation, image communication has greatly improved by the development of process- ing and transmission schemes supporting Regions-of-Interest (RoI) and Levels-of-Detail (LoD).7 Not violating the general requirements of scalable compression, most important information is placed at the beginning of the stream followed by less important data. This leads to an interest-ordered data-stream and allows to decode and view important image regions earlier than others. Any valid stream termination leads to an image where impor- tant regions are presented exclusively or at higher quality than others. Dependent on the option to interactively define or parameterize their behavior during streaming, there are two main RoI types: static6 and dynamic RoIs.8 With a static RoI the streaming order is predetermined at compression time, while a dynamic RoI allows re-ordering (usually within limits) of the data stream upon decompression. Due to their ability to adapt to changing demands and the implemented paradigm - Compress Once: Decompress Many Ways9 - dynamic RoIs are of exceptional importance in interactive environments. Besides its more technical related objectives, progression has also been applied to support semantical aspects of image communication. It has been shown1 that a well-designed, RoI-based refinement strategy can significantly improve the conveyance of selected image contents. Thus, progression can also be seen as a novel kind of information display combining a predefined or interactive animation of data contents with a highly resource- saving processing and transmission system. Aside from overcoming bandwidth limitations for raster imagery, it has also been applied to other kinds of system constraints10 and data.11 Its application to geometric data in particular has been researched for a considerable time as well. Examples include approaches of Hoppe12 and Lee et al.13 providing means to simplify, respectively, large triangle meshes and complex iso-surfaces. Here, the resulting simplification hierarchy allows to reconstruct the object in various LoD to meet rendering hardware and/or fidelity requirements. However aspects of transmission, especially the use of common formats, were secondary to all of these approaches. One application domain that has stimulated research in progressive vector transmission schemes are web-based GIS applications.14 Here too, initial development focused on the delivery of spatial data as raster images, which could be reduced by the use of established image progression schemes, e.g., the one proposed by Rauschenbach and Schumann.7 However, the desire to have access to more functionality, specifically direct querying and manipulation of geographic objects, soon stimulated exploration of communication protocols based on vector formats. The need to provide progressive refinement of vector maps to address bandwidth and device limitations has been recognized early. Bertolotto and Egenhofer15 proposed a general framework for progressive vector transmission based on cartographic generalization, but did not consider encoding into a specific data format such as SVG. Similarly, Yang et al.14 developed a hierarchical model for vector map data that facilitated progressive streaming of vertex data with client-side geometry reconstruction. Another frequent approach is to generate adapted maps at server-side according to request type and/or client profile;16 “progression” here being understood as the transmission of maps with increasing details as the user selects successively smaller map sections, which however does incur redundancy in transmission. An example for this method is the scheme introduced by Costa et al.,17 which uses SVG to encode adapted maps for transmission. However, all these approaches discuss challenges and concepts primarily from the perspective of cartographic generalization, i.e., preservation of topological consistency and the complexity of real-time generalization. Pro- gression is either based on proprietary data formats, or utilizes SVG merely to encode successive but self-contained maps for transmission. Only recently, research efforts have been directed at obtaining progression schemes for SVG that would allow incremental refinement without incurring redundant transmission. These approaches exploit the fact that SVG provides means to refer to external resources, thus allowing to fragment a vector graphics across several files. Li and Deng18 utilize this for a multi-resolution mechanism for SVG-encoded maps based on a hierarchy of “SVG collections” each comprised of a base and one or more detail layers. Client-side scripting is employed to insert references to detail layers on request, which automatically triggers file streaming and updating visuals. However, their approach is inherently bound to the thematic layers provided by the backing GIS server. It does also not support dynamic RoIs to control what detail is shown where, which we consider a crucial property in mobile image communication. In summary, progressive image communication based on raster data is well researched, but work towards progression schemes for vector-based graphics so far has mainly focussed on purely technical aspects neglecting compliant implementations. This is a limiting factor in communication scenarios that by principle require an agreement on protocol and data formats.
3. PROGRESSION SCHEMES FOR SCALABLE VECTOR GRAPHICS The aim of our approach is to make all benefits that progressive data communication can provide – previews, efficient data compression and transmission, Tours-through-the-data, and support for interactive systems – avail- able for arbitrary graphical content, as technical illustrations or visualizations, in vector format. In particular, we want to provide these features in a compliant manner for SVG as a data and transmission format with broad acceptance. Conceptually, a dynamic progression scheme comprises the following steps (for a more detailed discussion, we refer to Rosenbaum and Schumann19): (i) creation and encoding of a (LoD) hierarchy on geometry at server side; (ii) flexible server-side sequencing of the hierarchy to facilitate dynamic (interactive) RoI functionality and (iii) transfer (streaming) to the client; (iv) identification of the individual chunks to allow (v) decoding and partial reconstruction of the image at client side. Figure 2 illustrates this general process. With our approach, the SVG format is used to represent the hierarchy of the geometry required to apply progression to graphical contents. Following the general principle of scalable image coding, our objective is to re-order and stream this hierarchy in a way that decoding of a truncated stream leads to partially reconstructed content. In particular, the reconstruction of a truncated, re-ordered data stream is a fully compliant SVG image that can be processed and viewed by any SVG-capable tool. In correspondence to the steps outlined above, therefore, two requirements exist at server-side: (1) the SVG must be structured so that its primitive hierarchy allows the extraction of different LoD and (2) organized in a way that permits transfer of primitive data for arbitrary RoIs. On client side it must be ensured that (3) received Figure 2. A data presentation scheme using progressive refinement for the scalable display of vector data (on the basis of Rosenbaum and Schumann19).
SVG fragments can be identified and re-integrated to form a single, standard-conformal file that describes the graphical content transferred up to that point. After describing the fundamentals of the SVG imaging standard, we introduce two strategies that can achieve these objectives: (i) progression using linked files based on a decomposition of the original SVG into several linked files contributing to different levels of detail, and (ii) element-wise progression by dynamically decomposing the input file into its individual XML elements, whereas primitives are sorted according to semantic criteria. Due to constraints of the standard, each of the two strategies has its own advantages and drawbacks – and thus, application domains – in terms of transmission efficiency and client-side complexity. We provide a more detailed discussion of both strategies and its respective application domain in the following.
3.1 Fundamentals of the SVG imaging standard Scalable Vector Graphics (SVG) is a modularized language for describing two-dimensional graphics in XML, endorsed by the WWW consortium.2 SVG uses three types of graphic objects or primitives: geometric shapes (e.g., rectangles, circles, or paths), raster images, and text. Graphic primitives can be transformed, grouped, and styled by assigning presentation attributes like fill and stroke colors. Groups and individual objects are orga- nized in a hierarchy controlling how graphical elements are composed during rendering. SVG has a well-defined rendering model, which is mandatory for compliant user agents (UA). SVG drawings can be interactive and dy- namic. Animations can be defined and triggered either declaratively, i.e., by embedding SVG animation elements in SVG content, or via scripting. A rich set of event handlers can be assigned to any SVG graphic primitive. A supplemental scripting language further allows to modify the SVG Document Object Model (DOM), which provides complete access to all elements, attributes and properties. This facilitates sophisticated applications based on SVG. SVG has been explicitly designed to offer broad interoperability with other web technologies and standards. For example, Cascading Stylesheets (CSS) can be used for styling. In particular, it offers extensive support Figure 3. Progression scheme based on linked resource files (SVG fragments). The number of LoD corresponds to the fragment count. The client’s SVG UA resource resolving mechanism automatically re-integrates fragments into the images as they are received in progression order. The right column shows an example of a jet engine compressor divided into its skeleton and four LoD fragments. for integration with domain-specific languages (DSL) through XML namespaces20 and linked resources, like stylesheets or embedded images, based on the XML linking language21 (XLink). To enhance scalability of SVG in the face of linked resources, the standard recommends that user agents optionally support what is called “progressive rendering”.2 This, however, is much more limited than the progression schemes proposed here in that it only addresses asynchronism of client-side DOM processing: the rendering process is either allowed to proceed by skipping unresolved nodes, i.e., with references only partially loaded or in error, or halted temporarily. The draft stipulates that rendering should be automatically updated or resumed, respectively, as soon as any incomplete node becomes available. While this automatism eases the adoption of more sophisticated progression schemes by accepting incomplete SVG files and managing rendering updates, in and of itself this mechanism does not account for the superordinate requirements of image com- munication as outlined in Section 2. In particular, SVG’s progressive rendering has no notion of LoD, RoI, or re-ordering of data streams.
3.2 Progression using linked files The first strategy exploits SVG’s ability to incorporate linked external resources. Splitting up content over several linked SVG files is also referred to as SVG fragmentation.18 For the purposes of progressive transmission of SVG files, the object hierarchy of an input SVG file is re-organized to represent the required LoD hierarchy as content fragments. Fragmentation involves a two-stage process performed once per input file in a pre-process (cf. Fig. 3). As the primary decomposition we introduce four higher-order fragment types to reflect the principle components inherent to SVG files: structure, geometry, styles and scripting/animation. The secondary decomposition further subdivides geometry and styling attributes into LoD. The first fragment comprises all information required to describe the structure of a minimal valid SVG file. At the very least, this includes the XML header, SVG meta-data elements, the