sign in sign up
<<
Home , Technology

Proc. Natl Acad. Sci. USA Vol. 78, No. 11, pp. 6927-6931, November 1981 Cell

Computer-enhanced video microscopy: Digitally processed images can be produced in real time (image processing/phase-contrast optics/differential interference-contrast optics/tissue ) ROBERT J. WALTER AND MICHAEL W. BERNS Department of Developmental and Cell Biology, of California, Irvine, California 92717 Communicated by Keith R. Porter, July 21, 1981 ABSTRACT Digital processing techniques can be used to development of techniques for enhancing differential interfer- greatly enhance the available information in an optical image. ence-contrast (DIC) and polarizing microscope images (5-7) Although this technology has been routinely used in many fields may be the beginnings ofareal revolution in optical microscopy. for a number ofyears, little application ofdigital image-processing The most sophisticated enhancement and analysis of micro- techniques have been made toward analysis and enhancement of scope images is obtained through the use ofdigital processing the types ofimages seen most often by the biologist. We techniques on a previously digitized image. This digital image describe here a -based video microscope that is can originate from either a converted video signal, a photodiode capable of performing extensive manipulation and enhancement scanner, or an optical multichannel analyzer. Processing rou- of microscope images in real time. The types of manipulations possible with these techniques greatly surpass the enhancement tines such as image subtraction, edge enhancement, spatial fil- capabilities ofphotographic or video techniques alone. The speed tering, and contrast enhancement can be performed on a dig- and flexibility of this system enables experimental manipulation itized image in ways that far surpass the types ofmanipulations of the microscopic specimen based on its live processed image. possible with photographic or video techniques alone. These features greatly extend the power and versatility ofthe light Although digital image processing techniques have been ap- microscope. plied to biomedical problems for more than 20 years (8, 9), the primary emphasis of this has been toward of The need to record and analyze poor-quality, low-contrast mi- routine clinical work such as the classification of cell types, croscope images is a frequent problem confronting the cell bi- screening ofhistological samples, or preparation ofkaryotypes ologist. Although contemporary investigators have available to (10-14). Relatively little application of digital processing tech- them a wide variety of optical techniques (phase contrast, po- nology has been made toward enhancement oflive microscope larization, dark field, interference contrast, etc.) even the most images of the type most often seen by the research biologist. careful use of these techniques may still fail to produce a sat- Although several different microscope image analyzers are com- isfactory image from a given specimen. These image limitations mercially available, these have been designed pri- are particularly severe when the microscope is used as a marily for maximum speed when performing routines with di- for actual experimentation. Microirradiation or micromanipu- rect clinical applications, with a subsequent loss of speed and lation ofmaterial viewed through the microscope can be so re- versatility when they are programmed for other tasks (15). This stricted by image quality that an otherwise feasible experiment fact, in addition to the cost and of the equipment, can be rendered impossible. has generally inhibited the application of digital image-pro- Optical images that are unsatisfactory in their raw form may cessing techniques to real-time microscopy in basic biological still contain useful information that is unseen due to perceptual research. limitations of the eye. This hidden information can be We report here the development of a versatile computer- revealed through the use of various image enhancement tech- based video microscope system that is capable of performing niques that range in complexity from simple photographic pro- extensive and sophisticated image-enhancement routines on cessing to sophisticated digital processing routines. In recent microscope images in real time. The images produced by this years video technology also has been applied to light microscopy system appear to have better contrast and resolution than those as a method for routinely enhancing microscope images (1, 2). produced by conventional microscopy. This system is flexible Video with different light-transfer properties can be and interactive with the microscopist. Examples are presented selected for agiven application in the same manneras one would ofcontrast enhancement, edge enhancement, and background select a photographic , with the additional advantage that subtraction of microscope images produced from a live video a video will also produce an enhanced image in real signal by using digital techniques. time. Sophisticated electronic processing can also be performed on a video signal to reduce noise, subtract background, or make MATERIALS AND METHODS quantitative . Highly light-sensitive video - eras can be used to produce bright images from very weakly A schematic diagram of the image-processing system is shown illuminated microscope specimens, enabling continuous obser- in Fig. 1. vation ofliving material that is adversely affected by the usual Optics. Optical images are produced by a Zeiss AXIOMAT high level of illumination (3). Although this technology is still microscope in the inverted mode. Specimens can be viewed new, recent advances in video microscopy, such as the detection under phase contrast, polarization, differential interference of transient low-level subcellular fluorescence (2, 4), and the contrast, and other optical . The microscope controls include a 0.5-Am-step, XY-motorized, stage translational con- The publication costs ofthis article were defrayed in part bypage charge payment. This article must therefore be hereby marked "advertise- Abbreviations: ITF, intensity transformation function; DIC, differential ment" in accordance with 18 U. S. C. §1734 solely to indicate this fact. interference contrast. 6927 Downloaded by guest on September 27, 2021 6928 Cell Biology: Walter and Berns Proc. Nad Acad. Sci. USA 78 (1981) signal. Each ALU can produce a full 262,144-pixel image in one video frame time. Additional hardware features provide the capability for defining regions of interest in the image, substi- tuting image intensity values using predefined look-up tables, spatially offsetting images in memory, and manipulating the display of individual memory channels. Processed images are reconverted to video format and routed simultaneously to a 19- inch (48-cm) color monitor and the video tape recorder. Any combination of these manipulations can be performed on the live video signal in one frame time. LSI-11/23 Processor Controller. The image processor is a LSI/1 I DUAL dedicated processor that continuously ma- MICROCOMPUTER FLOPPY DISKS nipulates and displays the data within memory in response to the preprogrammed settings ofapproximately 250 16-bit control FIG. 1. Schematic diagram of the image-processing system. Im- registers. The LSI-11/23 (Digital Equipment, ages from the AXIOMAT microscope are converted to video format, Maynard, MA) controls the operation of the image processor then routed simultaneously to a black and white monitor and the dig- itizer of the image processor. Operation of the image processor is con- by performing read/write operations to these control registers trolled by the LSI-11 microcomputer using data and programs stored and controlling processor timing. FORTRAN and MACRO lan- on floppy disks. Processed images are reconverted to video format and guage programs are stored on floppy disks or an RL01 hard copy viewed on a color monitor. Live video images can be stored on the video disk. Digitized images can be stored on the floppy disks at the tape recorder (VTR) either before or after digital processing. DAC, dig- rate oftwo per disk. Complex image manipulations that cannot ital to analogue converter. be performed efficiently in the image processor alone can be accomplished in the memory ofthe LSI-11 by transferring small trol and an automatic autofocusing device that operates off of (8000 pixel) sections ofthe image to the LSI-11. However, these the video signal. Both devices can be controlled by the com- manipulations are slower than the image processor alone by 3 puter and image processor. Microscope images can be projected or 4 orders of magnitude. onto the video camera and viewed through the oculars simultaneously. Video Camera. The video camera (model LST-1, Sierra Sci- RESULTS entific Products, Mountain View, CA) consists of a 1-inch (2.5- Contrast Manipulation and Enhancement. A photograph of cm) chalcogenide (Newvicon) video tube plus rack-mounted a PTK2 tissue culture cell taken with phase-contrast optics is control . The sensitivity of this tube is somewhat shown in Fig. 2A. This image was photographed directly from between that of a silicon intensified-target camera and a stan- the video monitor and has not been digitally enhanced. A com- dard vidicon. This camera system has been selected for minimal puter generated histogram of the distribution of grey values geometric and shading distortion in the video signal. The scan- within this image is shown in Fig. 2B. This histogram shows that ning interval at the primary image is several times smaller than all of the grey tones in this image are tightly clustered around the resolution of the microscope at all magnifications. The out- a sharp central peak corresponding to the phase-neutral back- put video signal can be optimized for each image with external ground, and that many possible grey tones are not represented controls for gain, (dc) offset, and contrast. The in the image. This tight distribution is typical of images pro- video signal is generated in a 525-line format at a rate of 30 duced with phase-contrast or DIC optics. The shoulders of this frames per sec. Output from the camera can be directed si- histogram are produced by the phase-light and phase-dark re- multaneously to a 13-inch (33-cm) black and white monitor gions of the cell and contain all of the useful information in the (where it is viewed unaltered), to the input of the image pro- image. The narrow range of grey values in this image and the cessor, and to a video tape recorder. large amount of background result in a low contrast that makes Digitizer. The video digitizer converts each video line into it difficult to perceive fine detail. This condition is a result of 512 sequential eight-bit intensity values. Only the six most sig- the fact that any pixel in the image has a high probability of nificant bits of each discrete picture element (termed a "pixel") having neighboring pixels with grey values equal to or nearly are actually digitized per video frame; however, a true eight-bit equal to its own; consequently, there is low contrast between image can be produced in four video frame times by using cu- neighboring pixels, and much detail is hidden. mulative averaging techniques. Only the first 512 video lines This masking of detail can be overcome by arbitrarily reas- of each video frame are digitized; consequently, each original signing the grey values of the phase-light and phase-dark re- video frame is converted into a 512 X 512 array of pixels gions ofthe image to different grey values with higher contrast. (262,144 pixels total). Fig. 2D shows the image of Fig. 2A after the grey values were Image Processor. The image processor (model IP5500, reassigned in this manner. These grey reassignments are DeAnza Systems, San Jose, CA) was originally designed for specified by an intensity transformation function (ITF), derived analysis of LANDSAT images and has been specially by the image processor, that maximizes contrast in the desired adapted for use with the optical microscope. The image pro- region of the image. This function is analogous to the charac- cessor contains three separate 512 X 512 X 8-bit memory chan- teristic curve (sometimes called the Hurter-Driffield curve) nels that are configured to correspond to the red, green, and that is used to describe the light-transfer properties of photo- blue primary images of a normal color picture. Special hardware graphic film or video cameras. One of the advantages of digital allows this memory to store and display three different mono- enhancement techniques is that, unlike the light-transfer prop- chrome (black and white) images when a true color image is erties of and video tubes, this ITF does not have to be not desired. The processor itself is built around two eight-bit linear or even continuous. This extra capability can be used to arithmetic logic units that can each perform one of 96 different maximally enhance contrast in the information-containing re- arithmetic or logical operations on any combination of two in- gions of an image while leaving unaltered or even reducing the puts from the three-memory channels or the live digitized video contrast in background regions. The nonlinear ITF that maxi- Downloaded by guest on September 27, 2021 Cell Biology: Walter and Berns Proc. NatL Acad. Sci. USA 78 (1981) 6929

a!, 8 i Li 'U:-V'' 4

A V

FcIG. 2. Microscope images enhanced by using video and digital techniques. (A) Phase-contrast image of a PTK2 culture cell before digital en- hancement. (D) The same image after digital processing to enhance contrast in phase-light and -dark regions. (G) Same image as D, except the grey values of the phase-neutral regions have been arbitrarily assigned to zero (black). (B) Histogram of C showing the relative distribution of grey values. (E and H) Image histogram of D and G, respectively, after enhancement. (C, F, and I) Intensity transformation functions of A, D, and G, respectively. Note nonlinear of ITFs of I and H. (J) Differential image processed from the phase-contrast image of A. (K) DIC image of the same field of cells as in A. in J and K show the direction of optical shear. (Bar in A = 10 sm.) Downloaded by guest on September 27, 2021 6930 Cell Biology: Walter and Berns Proc. Nad Acad. SCti. USA 78 (1981)

mally enhances the contrast of the phase-light and phase-dark ject boundaries by subtracting offset images. The two offset im- regions of Fig. 2A, while simultaneously not affecting the con- ages are created by perpendicularly polarized beams of light trast ofthe phase-neutral background is shown in Fig. 2F. The that are retarded in phase in relation to one another. The two grey value histogram of the enhanced image is shown in Fig. images are subsequently "subtracted" when they are recom- 2E. Note that more detail of Fig. 2A is revealed after this pro- bined, and the retarded wavefronts optically interfere. A pho- cessing. A linear ITF that does not alter image contrast and is tograph ofthe same field ofcells taken with DIC optics is shown equivalent to a photographic film with a degree of contrast (y) in Fig. 2K for comparison. of one is shown in Fig. 2C for comparison. Although the images in Fig. 2J and K appear similar in many Once the ITF has been derived, the grey value reassignments respects, there are fundamental differences in- the way the im- defined by this function can be loaded into the appropriate reg- ages are produced that can make the digital processing tech- isters of the image processor and used to enhance the contrast nique more desirable than DIC microscopy in many circum- ofeach subsequent video image in real time as it is being written stances. Both techniques produce images that are approximations to processor memory. of the phase gradient in the original specimen (18). However, Pseudocolor Contrast Enhancement. When all possible grey specimens that are birefringent or optically rotate polarized values are contained in an image, contrast cannot be enhanced light may seriously degrade DIC images but will not interfere within one range of grey values without reducing contrast over with phase-contrast images (19). An edge-enhanced image pro- another range. This drawback can be overcome by using the duced from a phase-contrast image under these circumstances technique of pseudocolor enhancement. will not be subject to this degradation. We have also found that In our system each color image is displayed as the combi- the artifactual "halo" produced by phase-contrast optics may nation of red, green, and blue primary images, each of which actually produce better-defined edges in the differential phase- can vary in intensity over a 256-step range. More than 16 million contrast image than in the DIC image. The halo tends to ex- different color combinations are possible. The 256 shades of aggerate the phase gradient at some of the boundaries within grey in a normal monochrome image represent a small subset an image by producing a bright ring on one side ofa boundary ofthis largerclass ofcolor combinations; consequently, an image and a dark ring on the other, with a sharp inflection in intensity that utilizes all ofthe possible grey values in a monochrome im- at the actual boundary (18). Because the "difference" technique age has many unused color values when considered in the color we use produces a partial derivative ofthe image intensity func- domain. Individual grey values in a monochrome image can be tion, this exaggerated gradient produces a sharp edge at the arbitrarily converted to new color intensity values without af- actual boundary in the difference image while somewhat re- fecting the contrast of remaining grey tones. This technique ducing the actual halo itself. only differs from other contrast-enhancing routines in that in- An intense mercury arc is required to produce a DIC image dividual grey values are assigned to new colors and not simply on the AXIOMAT microscope in order to compensate for light to new shades of grey. losses at the polarizers and Wollaston prisms. This intense Edge Enhancement. Preferential enhancement ofthe edges source is often damaging to living material. In comparison, a and boundaries within an image is a commonly used processing digitally processed differential phase image is produced with technique for detecting objects and suppressing background phase-contrast optics and low-level tungsten illumination that (16). Fig. 2G shows an edge-enhanced image of Fig. 2A and D. does not have this effect. Consequently, the digital technique In this image the contrast of the phase-light and -dark regions may be preferred for viewing living specimens, even when was enhanced as in Fig. 2D; however, the grey values ofphase- equivalent images are produced by both techniques, in order neutral background regions were arbitrarily assigned to zero to avoid damaging the specimen. (black). As with the previous examples, the nonlinear ITF that produces this image was loaded into the memory of the image processor, and all subsequent video frames were similarly pro- DISCUSSION cessed in real time. The nonlinear nature of digital processing makes it possible to A more sophisticated technique for producing edge-en- create images that generally cannot be obtained with photo- hanced images is shown in Fig. 2J. This image was produced graphic or video techniques alone. Not only can certain parts directly from the phase-contrast image of Fig. 2A in a two-step of an image be preferentially enhanced in relation to others, as process. First, the digitized image is written into two memory was shown in Fig. 2 A, D, and G, but this nonlinear can channels of the image processor so that there are identical im- also be used to compensate for other sources ofnonlinearity in ages in each channel. Second, the image in one channel is spa- the system (20). An ITF can be used to correct for a nonlinear tially offset a small distance in the horizontal direction (corre- digitizer or video camera response in such a way that intensity sponding to approximately 0.5 ,um in the image), and this offset in the processed image is proportional to the optical density in image is subtracted from the first image. A constant value of128 the original image. Nonlinearity in the optical images can also is also added to each pixel to avoid the production of negative be corrected by this technique. For example, it is known values. The resulting image is then displayed on the monitor that object intensity in a DIC or polarization image is a function as a monochrome image. This process can be repeated in pipe- ofthe sin' ofthe phase-retardation angle ofthe object in relation line for each subsequent video frame as it is presented to the background (18). An ITF can be used to substitute image to the processor, consequently the edge-enhanced image is pro- intensity values in such a way that intensity is instead a linear duced in real time. function ofretardation angle. After this processing, differences When small spatial offset values are used, the image pro- in phase retardation between objects can be directly measured duced by this manipulation approximates the partial derivative from the differences in intensity between them. ofthe image intensity function in the direction ofthe offset (17). We have presented here several examples of digitally pro- Areas within the original image with high spatial frequencies cessed microscope images that reveal more information than (such as edges and boundaries) will produce large nonzero de- was discernible in the unprocessed video image. The highly rivatives, whereas slowly varying background shading will be interactive nature ofour system, and the ability to do processing suppressed. in real time allows us to quickly select the optimum enhance- DIC microscopy is an optical technique that also detects ob- ment parameters for each image while the specimen is being Downloaded by guest on September 27, 2021 Cell Biology: Walter and- Berns Proc. NatL Acad. Sci. USA 78 (1981) 6931

viewed. This speed and flexibility in turn makes it possible to 8. Prewitt, J. M. S. & Mendelson, M. I. (1965) Ann. N.Y. Acad. Sci. do precise manipulation and quantification of material viewed 128, 1035-1053. through the microscope based on the processed image. These 9. Lipkin, L. E., Watt, W. C. & Kirsch, R. A. (1965) Ann. N.Y. Acad. Sci. 128, 984-1012. features greatly extend the power and versatility of the light 10. Olson, A. C., Larson, N. M. & Heckman, C. A. (1980) Proc. Natl microscope. Acad. Sci. USA 77, 1516-1520. 11. Bowie, J. E. & Young, I. T. (1977) Acta Cytologica 21, 739-746. 12. Bradbury, S. (1979)J. Microsc. (Oxford) 115, 137-150. This research has been supported by National Institutes of Health 13. Dunn, R. F., O'Leary, D. P. & Kumley, W. E. (1975)J. Microsc. Grants HL 15740, GM 23445, and RRO 1192 and by U.S. Air Force (Oxford) 105, 205-213. Office of Scientific Research Grant 80-0062. 14. Castleman, K. R., Melnyk, J., Frieden, H. J., Persinger, G. W. & Wall, R. J. (1976)J. Reprod. Med. 17, 53-57. 15. Preston, K. (1976) in Digital Processing of Biomedical Images, eds. Preston, K., Jr. & Onoe, M. (Plenum, New York), pp. 1. Willingham, M. C. & Pastan, I. (1978) Cell 13, 501-507. 47-58. 2. Rose, B. & Lowenstein, W. R. (1975) 109, 1204-1206. 16. Pratt, W. K. (1978) Digital Image Processing (, New York), 3. Berns, M. W. (1974) Biological Microirradiation (Prentice-Hall, p. 480. Englewood Cliffs, NJ), p. 8. 17. Thomas, G. B., Jr. (1968) Calculus and Analytical Geometry 4. Schlessinger, J., Shecter, Y., Willingham, M. C. & Pastan, I. (Addison-Wesley, Menlo Park, CA), 4th Ed., p. 498. (1978) Proc. Natl Acad. Sci. USA 75, 2756-2663. 18. Allen, R. D., David, G. B. & Nomarski, G. (1969) Z. Wins. Mikr. 5. Inoue, S. (1981)J. Cell Biol 89, 346. 69, 193-236. 6. Allen, R. D., Travis, J. L., Allen, N. S. & Yilmaz, H. (1981) Cell 19. Lang, W. (1969) Zeiss Information No. 73, Reprint S41-210.2-5e Motility 1, 275-289. (Oberkochen, Federal Republic of Germany). 7. Allen, R. D., Allen, N. S. & Travis, J. L. (1981) Cell Motility 1, 20. Castleman, K. R. (1979) in Digital Image Processing (Prentice-Hall, 291-302. Englewood Cliffs, NJ), pp. 84-95. Downloaded by guest on September 27, 2021