© 2007, IS&T: The Society for Imaging Science and Technology

Ten Tips for Maintaining Digital Quality

Peter D. Burns and Don Williams, Eastman Company, Rochester NY

by system image quality requirements, physical imaging Abstract parameters provide the link to technology selection and The development of digital collections has benefited from interpretation. For example, image sharpness is a visual both the technology and price-volume advantages of the wider impression; MTF is a physical imaging parameter related to markets. With the advent of web-based delivery, image sharpness, and to optical design and adjustment (the standard file formats, and , however, have come a host of Technology Variables, of Ref. 3). technology related choices and concerns about variability and compatibility. We suggest ten simple principles that can be used to 3. Sampling is Not Resolution increase utility and reduce variation of the content Image sampling indicates the sampling interval between for most collection projects. on a particular plane in the scene (), or on the object (scanner). It is often referred to in pixels/mm or pixels/inch. Image Introduction resolution, or limiting resolution, has its roots in continuous optical The following are a collection of tips aimed at providing imaging systems. Resolution refers to the ability of an imaging guidance for those responsible for establishing and maintaining the component or system to distinguish finely spaced details. For imaging performance during image conversion projects. Each of (sampled) digital , the sampling rate (or mega- rating the suggestions is well understood (and often debated) in the for a camera) sets an upper limit for the capture of image detail. technical imaging community, but not always clear to the wider The size and architecture usually determine the group of clientele in museums, libraries, and similar institutions. image sampling, while other components also influence the Presented as short recommendations, they can form the basis for delivered image sharpness and resolution. questions that project managers can ask their technical staff and To understand this, consider a with a lens that imaging service providers. is well focused on a subject. After capturing the scene, the lens focus position is changed so the subject is not well focused. If we 1. Scan but Verify compare the two image files, they will have same image sampling Perhaps the most important function to include in a digitizing (and number of pixels), but the second digital image will appear workflow is a method and practice of verifying delivered imaging less sharp when viewed on a computer monitor or print. The performance. This applies whether comparing different scanners, second image will have lower resolution. or selecting service providers. Is the requested sampling rate (i.e. A well-established method for evaluating camera or scanner pixels per inch) actually being delivered? Is the resolution was developed for the standard, ISO 12233. Originally consistent with the sampling rate? To what extent does the developed for digital , the method has been applied to film encoding differ from a desired specification? All of these questions and print scanners, and CRT displays.4 Figure 1 shows the results call for establishing measurable goals for the imaging unit, when the above method was applied to this digital camera focus whether in-house or subcontracted. More importantly they serve as experiment. The resulting spatial Frequency Response (SFR) is a the basis for both real-time production control and acceptance function of spatial frequency. The lower dashed curve indicates the auditing; 1,2 in other words, good quality control. Remember, just effective loss of image detail information, e.g. limiting resolution, because content is digital does not make it error free. due to optical defocus. Performance verification measures can also be adapted to improvement initiatives. Once a performance measurement system is in place, however, questions will naturally arise as to the interpretation of results and possible corrective actions. The following suggestions and observations are intended to help in this area.

2. Imaging Performance is not Image Quality A common objective for image acquisition in cultural heritage projects is to produce high quality images. As a goal this statement may be adequate, but it is generally not helpful when evaluating how well an imaging function is being accomplished, and how to improve the system. In addition, overall image quality is usually understood to mean the visual impression, and this can have many components such as sharpness and colorfulness, and personal preferences. In this report we refer to imaging Figure 1: SFR measurement results for the digital camera with two lens performance as described by physical imaging parameters.3 While focus positions the particular acceptable levels of performance will usually be set

Proc. IS&T Archiving Conf., pg. 16-22, IS&T, 2007

4. Single Stimulus Visual Assessments of Color, This can be easily understood based on the SFR plot, Tone Can be Unreliable computed using an edge feature in each digital image. It has been Work in the imaging field for some time and one is bound to suggested that the spatial frequency at which the SFR is reduced to see examples of how easily we can be fooled into perceiving 0.1 (10%) serves as a measure of limiting resolution. The arrows significant color and brightness differences in natural scenes of indicate these (relative) values, which are consistent with the what are otherwise physically identical stimuli. Most of these corresponding clock faces. A different measure, acutance, however illusions are due to the complex spatio-temporal adaptation has been used to predict image sharpness. This integrated responses of the human visual system.5 An example is shown in frequency-weighted measure7 will be a function of viewing Fig. 2 from a collection 6 that could provide hours of rainy-day distance. When viewed as presented here, however, the higher fun. response at the lower frequencies indicates a higher sharpness for the upper picture.

Figure 2: Example from Ref. 6. While the center regions of the top row are perceived as having different contrasts, masking the surround (bottom row) reveals they are of identical.

While these effects are real in terms of perceived brightness, they are misleading when interpreted as radiometric differences. Viewing environment, observer variability and preferences are major factors. For system evaluation, dual stimuli visual assessments under the same viewing conditions are advisable. To a lesser extent visual evaluations of image microstructure should also be treated cautiously. For example, visual resolution judgments can be driven by feature contrast: the higher the high sharpness, low resolution contrast, the higher the resolution rating. Another problem, well 1.0 known in traditional , is that the size, shape and extent low sharpness, high resolution of the target features can bias evaluations. When using the classical bar target patterns for limiting resolution, the SFR viewer has a priori knowledge of the number of bars to be 0.5 detected. This can bias the evaluation. Results can also vary with length of the bar, where a longer length feature can result in a higher estimated resolution value. While on the topic of limiting resolution, we also consider 0.1 overall image sharpness. Figure 3 shows two versions of a railway 0 station scene. The upper picture resulted from applying digital 0.25 0.5 sharpening filter to an unsharp digital image. The lower version Frequency , cy/pixel was captured with a well-focused camera without digital sharpening. The limiting resolution, as illustrated by the expanded Figure 3: Upper picture is for unsharp capture followed by digital clock face, is higher for the lower picture, as we might expect. The sharpening. The lower is for a well-focused capture without sharpening. The graph shows the corresponding SFR results based on analysis of digitally sharpened version, however, gives the impression of a an edge feature in each picture. sharper image.

5. Visual Assessments of Spatial Artifacts Can be resolution limit Reliable optical resolution due to poor A good skill for detecting ill-behaved image performance of limit CFA scanned images is visual literacy. An abridged definition, taken reconstruction from Brill, Kim and Branch8 and adapted for the evaluation of digital image performance is,

Visual Literacy: Acquired competencies for interpreting and composing visible messages. A visually literate person is able to interpret visible objects to detect unnatural or unexpected behavior, and identify possible underlying sources of performance variation.

As indicated in the previous section, interpretation of single stimulus visual assessments for color and tone reproduction evaluation can be unreliable. Doing so for spatial artifacts of both and noise, however, can be dependable and valuable. This is because spatial and noise artifacts associated with digital imaging often occur in relatively local regions. When displayed, several suspect regions are within the same visual field, and many variations, e.g. directional differences, in characteristics can easily be detected. In effect, several stimuli can be presented to the eye and evaluated nearly simultaneously. Concurrent viewing improves the chance of detecting spatial variation, but this alone may be insufficient. As the definition indicates, some level of image level discrimination and knowledge of expected behavior is Figure 4: Example of normal optical degradation in resolution (left) necessary. compared to degradation caused by , and color reconstruction A classic example of such behavior is JPEG blocking (right). Notice checkerboard pattern of right image artifacts. A reasonably initiated viewer can detect such artifacts in displayed images. The important observation here is that 8x8 pixel blocks are not naturally occurring in images. One improvement of the more recent JPEG 2000 method is the replacement of the fixed 8x8 or 16x16 pixel blocks with more natural looking textures. Another case for visual inspection is the detection of objectionable artifacts from color reconstruction errors for digital cameras employing a (CFA). Figure 4 illustrates an example of the way these reconstruction errors manifest themselves with an unexpected colored checkerboard pattern. Digital images convey image information by sampling the continuous optical image at the sensor and reconstructing (a version of) it at the display or . The left-hand picture resulted from well-matched optical quality of the lens and sampling at the sensor. The poor reconstruction, which includes color-interpolation, evident in the right-hand picture results from insufficient sampling (low-resolution) of the optical image, i.e. aliasing. Figure 5: Example of non-uniform noise texture due to spatial Large uniform areas are particularly suited in aiding the interpolation processing. The contrast has been enhanced here. visual detection of suspicious imaging behavior. As uniform areas, one expects the noise and textures to be rendered uniformly. An example where noise texture varies in a periodic and predictable 6. Raw Files, Be Careful What You Ask For fashion is shown of Fig. 5. One should see a grid-like pattern Recently there has been interest in acquiring digital images as defined by high and low noise textures in the otherwise uniform so-called raw files. Professional can use raw files to area. The automated detection of such characteristics calls for avoid some of the camera manufacturer’s image processing, e.g., analysis software tools beyond those routinely applied for quantization. Others make the case for an open raw image performance standards, but available by skilled visual inspection. file format that would extend the life of archival digital content. Before embracing this approach for collections, however, we should consider the meaning of the data contained in such files. We do this by tracing the state of data as it travels from object to digital image file, as outlined in Fig. 6.

Sensor Data Sensor defect, uniformity, Raw corrected Original Scene Sensor correction CFA Data

Raw color De-mosaicing white balance, interpolated Data camera color correction

Scene referred Color rendering: gamut Output referred Color encoding mapping, preference color encoding

Display transform Softcopy image Output referred Color encoding

Printer Hardcopy image transform

Figure 6: Example color image processing path. Adapted from information from K. Parulski (Eastman Kodak) and J. Holm (HP)

The reflected, or transmitted light from the object is collected is to use a simple test method for the detection of several sources by the optics and detected by an image sensor. The detected data of performance variation. may then be processed for sensor defects and uniformity. Once an image acquisition system is assembled and aim or If the imager used a color filter array, the result is a single matrix acceptable performance is established, the routine measurement of corresponding to a spatial pattern of repeated, e.g., red, green and the neutral tone-response curve, also known as the electro-optic blue, signals. At this point these raw data constitute the first form conversion function (OECF), can be used to evaluate performance of ‘raw’ recorded image, the raw corrected CFA data. deviation. This is accomplished by including a series of gray steps The next step in a typical processing path is the generation of in a test target, as shown in Fig. 7. a fully populated three-color image array. Propriety algorithms, Figure 8 shows the result of this measurement for a digital aimed at minimizing color artifacts, can be applied here. This de- camera. The aim response was for equal red, green and blue levels mosaicing operation is the interpolation of the single-record array for each of the gray steps. Note that the general shape of the to a ‘raw’ interpolated red, green and blue data set. response indicates a rendering for electronic display, but there is a White-balance, and matrix color-correction operations are red cast to the acquired image file. The higher red signal levels in usually applied next. The result is an image data set that is in a the mid-range indicate this. While this test does not differentiate scene-referred color encoding. This version of raw image data may all causes of tone- and color reproduction, it does provide a simple be the most useful for archiving purposes. Note, however, that this indicator of performance problems. is not the form in which most digital images are currently delivered. 8. Digital Cameras do not Produce The final step in the image processing chain is the rendering, For digital image conversion projects a common request is for usually for display. The result is a finished image data array in an a method to measure color accuracy or color reproduction. This out-referred color encoding. This step may be a simple color-space might appear to be a simple matter - we have all heard of using transformation, but can also include choices for gamut mapping CIE ΔEab values for color differences. Digital images, and their and color preference. digital pixel values, however, are not physical measurements and While the above steps are common in color image acquisition certainly not color stimuli. It is only when we interpret these systems, specific implementation details will vary. Understanding values, and their variation, in terms of the physical characteristics the signal (color) encoding of a raw image is as important as of the input scene or object that we can ascribe a color fidelity agreement on a particular file format. measure to our image acquisition process. Interpreting the digital signal encoding that was used for a 7. Question of Balance: Keep the Neutrals Neutral digital image is a first step in using a digital image, e.g. displaying Most technical publications on imaging performance, color on a monitor, or evaluating the accuracy of the image acquisition image quality, and image processing address the design of the system. This is currently done in systems using hardware and software elements, rather than system control and the ICC color profiles, which specify the intended encoding of the maintenance. Since we are focus here on recommendations for digital image. Such systems offer clear advantages for color evaluating and maintaining imaging performance, our suggestion communications in both open and closed color imaging systems.

However, most digital images currently exchanged, those on Elements of this test target can be used to evaluate several sites, are not color managed. In stead a color encoding is imaging performance variables. One of which, color image assumed by the user’s software as part of image display. Thus capture, can be based a colorimetric interpretation as shown in Fig. sRGB is probably the most common interpretation of posted image 9. The first step is to identify regions of interest (ROIs) in the content. digital image that correspond to specific target elements. From these areas their pixel values can then be used to estimate corresponding statistics. In this case, the mean (average) red, green and blue values for each color patch are computed. The color encoding of the digital image that is specified or assumed is then used to transform these data to an equivalent set of CIELAB coordinates in the third step. These test values can then be compared with aim CIELAB coordinates in the next step of the procedure. Color-difference measures, such as ΔEab or ΔE00 (CIE ΔE2000), are commonly used.

Find target ROIs Compute mean Transform pixel in image array RGB values for ROIs RGB to CIELAB

Compute difference Compare with Aim target measure process limits CIELB values

Figure 9: Outline of method for evaluating color capture based on CIELAB color differences. ROIs are the digital regions corresponding to several color target areas.

9. Consider Both Intra- and Inter-Image Variation It is natural to investigate imaging performance that varies Figure 7: Image-level target beside source document, from Ref. 1. Digital from one day to the next, or between sequential image files. Not image is courtesy of Cornell University. only is this consistent with common production quality assurance practice, but such differences may be evident when an exhibit is drawn from collections. Results from such periodic inter-image evaluation can be presented as control charts for important summary measures. In addition, we often observe considerable image variation across the digital image. For copy-stand cameras there is often variation center-to-edge signal uniformity, sharpness and spatial (barrel or pincushion) distortion. Several digital cameras used in cultural institutions apply image processing to compensate for optical distortion. This is helpful, e.g., for a if it is characterized at several lens positions. This form of optical distortion correction, however, results in a non-uniform (re)sampling of the correctly projected scene in the digital image. Alternatively, we can think of the as having a variable optical magnification (object-to-image) at the sensor. The solution is to resample the image array in a way that compensates for the Figure 8: Measured OECF showing a red cast in the midrange (red = *, non-uniform sampling. This image resampling requires green = o, blue = x). The line follows a curve fit to luminance-weighted interpolation and therefore some loss of spatial image detail. data. To simulate this effect, a scanned edge feature was placed in If interpretation of the color encoding is the first step in a larger digital image array at the center and corners, as shown in determining the color accuracy of image acquisition, how do we Fig. 10. Barrel distortion was introduced in software, with 5% measure performance? Having reference objects in the scene is effective optical distortion at the corners. This distorted image particularly helpful, whether these are part of an image-level or array was then corrected using the same software, so as to invert device-level target. Figure 7 shows an example image with an the first operation. Since both the original edge and the corrected image-level test target that contains elements for the evaluation of edges where straight, the primarily differences are due to the several performance measurements. image resampling which used bilinear interpolation. Note that the degree of loss in image detail, as indicated by a lower SFR in Fig.

11, is greater at the corner, as expected based on a larger effective With the development and refinement of digital imaging in interpolation ratio. Thus corrected images appeared less sharp near the mid-1990's there were few digital imaging standards, the corner than at the center when viewed, as the SFR results especially for performance. This lead to the use of various and predict. sundry methods for imaging resolution, color and noise Evaluating imaging performance across the image is performance, not based on vetted standards. For example, image important since hardware components can introduce variation. sampling in terms of pixel per inch was indistinguishable from true Image processing can compensate for some sources, but beware of optical resolution. Tone and color reproduction measurement were knock-on effects, such as blurring, or contouring. based solely on encoded digital values rather than the relation between those values and the physical characteristics of the source document itself. When was used for this purpose, its behavior, in practice, frequently did not follow accepted norms. And dynamic range was typically defined in terms of signal encoding (bit depth) rather than the appropriate signal and noise based metrics. Over the last ten years, members of the imaging community have developed several international standards for electronic imaging performance and quality. As noted in his overview paper, McDowell 9 aptly states that the greater advantage of these standards is to serve as valuable resources in the development of cooperative agreements where the interests of many different imaging parties converge. This is especially true for the cultural heritage community where no single institution is influential in Figure 10: Test image used for the barrel distortion correction promoting or dictating practices to the greater community. experiment, (shown here without distortion) Benefits of such standards include improved process control, informed technology selection and reduced costs. Benchmarking 1 and routine performance evaluation using tools enabled by no distortion 0.9 corrected 90% field international standards guidelines provide a path to daily workflow corrected center 0.8 monitoring. When the needs go beyond current established

0.7 standards, they can serve as starting points for adapted methods in libraries and museums, as is currently underway for other 0.6 applications. 10 0.5 SFR 0.4 Summary 0.3 If the purpose of computing is insight, not numbers, 11 so it is 0.2 for the techniques that we apply to the evaluation of digital 0.1 imaging performance. To this end, the above suggestions and 0 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 observations are aimed at providing technical guidance to those who seek to maintain or improve digital image collections. Frequency, cy/pixel

References Figure 11: SFR results from the distortion correction experiment [1] M. Stelmach and D. Williams, When Good Scanning Goes Bad: A Case for Enabling Statistical Process Control in Image Digitizing Workflows, Proc. Archiving Conf., IS&T, pg. 237-243, 2006. 10. Without Standards, You Do not Have a Legacy [2] R. J. Murray, Managing a Quality Practice in Cultural to Stand on Heritage Institutions: Statistical Quality Control Tools and Until about 1995 standard imaging practices were largely Techniques, Proc. IS&T Archiving Conference, pg. 96-104, 2006. dictated by a small number of companies who dominated [3] P. G. Engeldrum, Psychometric Scaling: A Toolkit for Imaging traditional image capture and printing through sound industry or Systems Development, Imcotek Press, Winchester, MA, 2000, ch.1. de-facto standards. There was interoperability for different [4] S. Triantaphillidou, and R. E. Jacobson, A Simple Method for manufacturers' films, with processing and printing, and imaging Measurement of Modulation Transfer Functions of Displays, Proc. performance was generally stable and understood. The language of PICS Conf., IS&T, pg. 139-144, 2000. imaging and communication between parties was unambiguous. [5] E. H. Adelson, Lightness Perception and Lightness Illusions, The New Cognitive Neurosciences, 2nd ed., M. Gazzaniga, ed.: MIT By applying those standards, one could rely on maintaining a Press, Cambridge, MA , pp. 339–351, 2000. connection between object and image. For instance, one can [6] http://www.michaelbach.de/ot/ confidently evaluate the optical provenance of a document based [7] E. M. Granger and K. N. Cupery, An Optical Merit Function (SQF), on standardized performance methods for exposure, resolution, Which Correlates with Subjective Image Judgments, Photographic and processing of the film used to capture it. Standards and Science and Engineering, vol. 16, pp. 221-230, 1973. available technical data provided a common language by which the objects’ legacy can be maintained.

[8] J. M. Brill, D. Kim and R.M. Branch, Exploring the visual future: art Author Biographies design, science and technology, The International Visual Literacy Peter Burns is with Eastman Kodak. His work includes medical and Association, Blacksburg, VA., pp. 9-15, 2001. dental image processing, and the development of imaging performance [9] D. Q. McDowell, Overview of the ANSI and International Standards evaluation, analysis and software tools. Previously he was with Xerox Process, Proc. SPIE-IS&T Electronic Imaging Symposium, SPIE vol. Corp. A frequent speaker at technical conferences, he is currently 5294, pp. 1-6, 2004. contributing to the I3A Phone Camera Image Quality initiative. He [10] D. Williams and P. D. Burns, Applying and Extending ISO/TC42 has taught several imaging courses: at Kodak, SPIE and IS&T technical Digital Camera Resolution Standards to Mobile Imaging Products, conferences, and at the Center for Imaging Science, RIT. Proc. SPIE-IS&T Electronic Imaging Symposium, SPIE vol. 6494, Don Williams is also with Eastman Kodak Company. His work 2007. focuses on quantitative signal and noise performance metrics for digital [11] R.W. Hamming, Numerical Methods for Scientists and Engineers, capture imaging devices and imaging fidelity issues. He co-leads the TC42 McGraw–Hill, New York, 1962. standardization efforts on digital print and resolution (ISO 16067-1, ISO 16067-2) scanner dynamic range (ISO 21550) and is the editor for the revision to digital camera resolution (ISO 12233). Don is also a member of the I3A Cell Phone Camera Image Quality initiative.