Signal Processing: Image Communication 17 (2002) 131–144 File format technology in JPEG 2000 enables flexible use of still and motion sequences J. Scott Houchina,*, David W. Singerb a Eastman Kodak Company, Rochester, NY, USA b Apple Computer, Cupertino, CA, USA Abstract While there exist many different image file formats, the JPEG committee concluded that none of those formats addressed a majority of the needs of tomorrow’s complicated imaging applications. Many formats do not provide sufficient flexibility for the intelligent storage and maintenance of metadata. Others are very restrictive in terms of colorspace specification. Others provide flexibility, but with a very high cost because of complexity. The JPEG 2000 file format addresses these concerns by combining a simple binary container with flexible metadata architecture and a useful yet simple mechanism for encoding the colorspace of an image. This paper describes the binary format, metadata architecture, and colorspace encoding architecture of the JPEG 2000 file format. It also shows how this format can be used as the basis for more advanced applications, such as the upcoming motion JPEG 2000 standard. r 2002 Elsevier Science B.V. All rights reserved. Keywords: JPEG 2000; JP2; File format; Metadata; XML; Color; ICC; Moving pictures; Streaming; MPEG 1. Introduction addition to the codestream syntax. Therefore, as part of the JPEG 2000 standard, the committee For several years, the JPEG committee (ISO/ has defined a pair of tightly linked standard file IEC JTC1/SC29/WG1) has been working on a formats. The minimal/simpler variation is called next generation image compression algorithm JP2 [6] and is targeted at applications that are called JPEG 2000 (ISO/IEC 15444-1).1 The limited to the storage of RGB and grayscale compression algorithm itself is wavelet based and images. The JPX [6] format expands the capabil- promises to provide many new image access ities of the standard to allow for the specification features to applications. To practically use those of other types of images (both photographic and features, however, the codestream must be com- non-photographic) and defines a set of standard bined with all of the information describing that metadata fields. The JPX file format also allows image; this requires the use of a file format in for the specification multiple images within a single file, and for the combination of those images *Corresponding author. Tel.: +1-716-588-8495. through compositing and animation. While using E-mail addresses: [email protected] (J.S. Houchin), [email protected] (D.W. Singer). these formats is optional for conformance, either 1 ISO/IEC 15444-1, Information technologyFJPEG 2000 applications are strongly encouraged to use the image coding system. JP2 or JPX file formats where appropriate. 0923-5965/02/$ - see front matter r 2002 Elsevier Science B.V. All rights reserved. PII: S 0923-5965(01)00023-6 132 J.S. Houchin, D.W. Singer / Signal Processing: Image Communication 17 (2002) 131–144 There are three main aspects of the JP2 and JPX important to define a compliance point at which file formats: all readers (including those targeted at the * binary container, consumer desktop) could read all files written to * colorspace specification, that compliance point. This compliance point is * metadata embedding. the JP2 file format. All readers are required to read As described in the following sections, the file all JP2 files. Applications that require features formats provide a strong foundation and feature greater than those defined as part of the JP2 must set for today’s applications. In addition, it is well use the JPX file format. understood that new applications are created every It is important to note, however, that many day, and that these file format must continue to aspects of the JPX file format are compatible meet the needs of tomorrow’s applications for with inclusion in JP2 file, and thus it is possible many years. As such, these formats are built upon (and appropriate in many situations) to create JPX a strong foundation with well-defined means for files that can be completely read by a simpler JP2 extension. This extension mechanism will also be reader. described. Note that at the time of writing, neither the JPEG 2000 standard nor the Motion JPEG 2000 3. Binary container standard have been finalized; therefore, aspects may be changed as the standards are finalized. The At the highest level, the JP2 and JPX file final standards, not this paper, should be taken as formats are simply binary containers. They pro- the definitive references. vide a mechanism to encapsulate both the com- pressed codestream and any metadata required to describe the decompressed data (such as the 2. Why two file formats? colorspace). However, the binary container also specifies how applications can gain access to the In a world that already has too many choices, it data embedded within the file. To truly meet the may come to the surprise of the developer that the needs of today’s and tomorrow’s applications, that JPEG 2000 standard defines not one, but two file binary container must provide quick access to the formats for use in ‘‘single-image’’ type applica- individual elements, yet not overburden applica- tions. The reason for this is very clear when one tions that desire to modify or add new elements to considers the state of the industry with respect to the file. the largest group of potential users: consumers. To provide a file format that is useful for a large number of target applications and industries, that 3.1. File structure file format would have to have a very large feature set. For example, commercial graphic arts applica- Conceptually, a JP2 file contains a sequence of tions require a format that can contain CMYK boxes. Each box encapsulates one element of the image data as well as the ICC profiles that describe file, such as the codestream or a group of metadata that data. Similarly, remote sensing applications fields. Many of those boxes directly contain the require support for many channels of image data. data embedded in the file. Other boxes, called The problem is that it is not practical to implement superboxes, just contain more boxes, providing a reader that can be used in every application area, only grouping and higher level meaning to other and thus there will be situations in which a pieces of data. The binary structure of a box is as particular JPEG 2000 file reader cannot open follows (Fig. 1). a particular JPEG 2000 file. While many industries The file is a sequence of boxes. Each box then have learned to deal with such interoper- contains header and contents fields. The header of ability problems, this type of situation cannot be the box starts with a 4-byte length followed tolerated on the consumer desktop. Thus, it was immediately by a 4-byte type. J.S. Houchin, D.W. Singer / Signal Processing: Image Communication 17 (2002) 131–144 133 JP2 file Box structure Box0 L: Box length (32 bits) Boxi T: Box type (32 bits) XL: Extended length (64 bits) Present only if L = 1 Boxk D: Box contents (variable length) Boxk0 L = 0 to end of superbox Box km Li = L = 1 XL L 8L Boxn Fig. 1. File and box structure. The length field of the box, L; specifies the Box 0 Box 1 Box 5 amount of data in the box, in bytes, including all box header fields. Because the length of the box Box 2 Box 3 Box 4 includes the length and type fields, the smallest L L L L Box 0 Box 2 Box 3 Box 4 L legal box size is 8 bytes. Values of the L field Box 5 smaller than 8 are used to indicate special cases. If LBox 1 the value of L is 0, then the box contains all data Fig. 2. Jumping from box to box. up to the end of the superbox, or to the end of the file for top-level boxes (boxes not contained within any other superbox). For example, if the value of L in Boxn in Fig. 1 were 0, then Boxn would go to Box 1 was not understood by a reader, it would the end of the file. If the value of L in Boxkm were not recognize the existence of Boxes 2–4 because 0, then Boxkm would go to the end of Boxk. they would be completely skipped by jumping the If the value of L is 1, the box may be longer than length of Box 1 from the beginning of Box 1. can be represented by a 4-byte field. In this case, This box structure is what provides for the the extended length field immediately follows the extension of the simple JP2 file format into more box-type field. This 8-byte field specifies the actual complex formats, such as the JPX file format. For length of the box. example, there are two additional file formats The type field specifies the kind of data found planned at this time, each building upon the within the contents of the box. Generally, box frameworks defined as part of the family: types are represented by four ASCII2 characters * JPM enables the storage of compound images rather than integer codes. using multiple compression types across a single Once the length of the box is determined, the scene. sequence of boxes in the file can be parsed, even if * Motion JPEG 2000 enables the storage of the format of the contents of the individual boxes sequences of individual image frames (such as is unknown. Fig. 2 shows how the box lengths can burst sequences from digital cameras or digital be used to jump from box to box.