Compression Evaluation of Surgery Video Recordings Retaining Diagnostic Credibility (Compression Evaluation of Surgery Video)

OPTO-ELECTRONICS REVIEW 16(4), 428–438

DOI: 10.2478/s11772-008-0041-0 Compression evaluation of surgery video recordings retaining diagnostic credibility (compression evaluation of surgery video)

M. DUPLAGA*1, M.I. LESZCZUK2, Z. PAPIR2, and A. PRZELASKOWSKI3

1Collegium Medicum, Jagiellonian University, 12 Œw. Anny Str., 31-008 Cracow, Poland 2Department of Telecommunications, AGH University of Science and Technology, 30 Mickiewicza Ave., 30-668 Cracow, Poland 3Institute of Radioelectronics, Military University of Technology, 2 Kaliskiego Str., 00-08 Warsaw, Poland

Wider dissemination of medical digital video libraries is affected by two correlated factors, resource effective content compression that directly influences its diagnostic credibility. It has been proved that it is possible to meet these contradictory requirements halfway for long-lasting and low motion surgery recordings at compression ratios close to 100 (bronchoscopic procedures were a case study investigated). As the main supporting assumption, it has been accepted that the content can be compressed as far as clinicians are not able to sense a loss of video diagnostic fidelity (a visually lossless compression). Different market codecs were inspected by means of the combined subjective and objective tests toward their usability in medical video libraries. Subjective tests involved a panel of clinicians who had to classify compressed bronchoscopic video content according to its quality under the bubble sort algorithm. For objective tests, two metrics (hybrid vector measure and hosaka Plots) were calculated frame by frame and averaged over a whole sequence.

Keywords: video, multimedia, video compression, quality evaluation, bronchoscopy.

1. Introduction sion techniques, Sect. 2 shows details of a research approach including subjective and objective tests, Sect. 3 pin- Research and development efforts of academic societies points the results obtained, and Sect. 4 concludes the work. and commercial companies in the area of digital video li- Related acknowledgements and references are given in the braries (DVLs) reveal a significant potential behind soft- end of the paper. ware and hardware platforms allowing catalogued digital video content to be searched for, retrieved and accessed 1.1. Bandwidth efficient access to medical digital over the Internet. video libraries There are several implementation problems pertinent to DVLs. With no doubt, the video compression standards are There are some applications in which DVLs appear to be mostly recognised and widespread. However, emerging especially useful. Tele-medicine is an example of a domain medical DVLs (MDVLs) add a new dimension as lossy where an MDVL can be used for storing and sharing nu- compression techniques to be used therein have to be both merous video sequences concerning examinations, opera- resource-effective and medically credible (despite inevita- tions and other medical procedures. Their later play-out can ble visual impairments due to compression). be done for educational or documentary purposes as well as The presented case study of recordings of bronchosco- to support computer aided clinical diagnostics (often re- pic procedures proves that it is possible to achieve nar- ferred to as medical images understanding) [1,2]. A cor- row-band video streams of long-lasting and low motion rectly configured library, storing unique content, allows for surgery procedures while keeping their diagnostic potential a remote and efficient access to voluminous multimedia re- intact. Thorough combined subjective and objective tests of sources. Moreover, an MDVL user does not have to pos- lossy video codecs revealed their maximum feasible com- sess any special device or software except a computer with pression ratios CR still suitable for diagnostic purposes. max an Internet browser and a video signal codec. The remainder of this paper is organised as follows: The development process of an MDVL should include Section 1 justifies a bandwidth efficient access to MDVLs all the steps necessary for a digital library preparation. as the research motivation, presents characteristics of bron- They cover conversion of a content from the physical/ana- choscopic recordings and state-of-the-art related compres- logue into the digital form, extraction or creation of meta- data or indexing information describing the content, stor- *e-mail: [email protected] age of digital content and meta-data in a multimedia reposi-

428 Opto-Electron. Rev., 16, no. 4, 2008 tory, establishment of client services for the browser, con- since 1998 have enabled video clip adjuncts being to be tent delivery via file transfer or streaming media, access added to papers published online [9]. Adding the video im- through a browser or dedicated client, as well as establish- aging improves the impact of training on health care pro- ing access via network. The implementation of an MDVL fessionals. Lavitan et al., in a group of paramedic trainees, locates itself within the efforts of development of medical evidenced the benefits of video viewing before undertaking virtual learning and diagnostic environment for the health practical interventions. The success rates among trainees professionals’ community. who, apart from traditional training courses, were given an Miron and Blumenthal described the approach to con- opportunity to see video of intubation, were higher than in vert an analogue video to digital data implemented in the a group of trainees not given such an opportunity [11]. surgical ophthalmology unit [3]. Leung et al. described the use of streaming audio and Digital video may be used in these areas of medical ed- video technology with the objective to enhance the emer- ucation where traditional approaches were not successful. gency physician’s education. In this case, lectures were re- Hamilton et al. explored the application of digital video in- corded with a digital camera recorder in a soundproof stu- tegrated with a computer-assisted learning system allowing dio. Both high and low bandwidth options were available for the study of post-natal human development. The behav- to optimise the viewer’s Internet connection. The iour patterns of children aged between 6 weeks and 12 RealPlayer software was used for playback. The authors months were recorded on a VHS tape and then digitalised, stressed the advantages of such an approach to medical ed- edited and integrated with a computer-assisted learning ucation which, relying on a flexible online learning experi- system which could be accessed through the network [4]. ence, allows the educational process to be realised at the The common feeling of many clinicians performing in- moment convenient to the clinicians undertaking the course vasive procedures, particularly surgeons, is the impression [12]. In some settings, “orientation video” is made avail- that many extremely interesting materials cannot be re- able through the Web in order to improve access and com- corded during every day activities and shown afterwards. pliance with the department of emergency for residents and Kumar and Pal described the cost-effective and time-sa- students who begin to train in a given unit [13]. ving approach to recording digital video during cardiac sur- The researchers from the University of Kentucky used gery procedures. They used a special video camera mount- streaming video technology to develop a webcast model to ing arm installed on an operating theatre [5]. allow institutions to broadcast live as well as pre-recorded The technology of the Internet streaming video re- surgeries, conferences and courses in real time over net- sources has been appreciated by medical educators who works. Gandsas et al. focused on the broadcast of pre-re- perceived it as a possibility to enhance the educational pro- corded laparoscopic para-esophageal hernia repair to na- cess [6]. tional and international audiences using desktop computers The results of the use of streaming video as an addi- enabled with off-the-shelf, streaming software and standard tional educational support were tested by Green et al [7]. hardware [14]. The use of medical digital video library is These authors evaluated the student nurses’ reactions to usually accompanied by other tools allowing for a discus- streaming videos supporting a life sciences module. The sion about course materials. Such an approach was applied students were able to view online e-learning sessions with by Wiecha et al. who enabled the student’s evaluation of an integrated streamed video. From the 656 students who the online modules to reinforce the course aims and con- entered e-learning resources online, more than half brow- cepts in discussion groups [13]. sed the video streams. From this group, about 60% appreci- The European Institute of Tele-surgery (EITS), founded ated viewing the videos, however, in fact, only 25% con- in 1994, supports the use of educational resources for sur- firmed that they were able to learn from them [7] (probably geons. The Institute was determined to develop new com- because of poor picture quality). The usefulness of online puter and training technologies in surgery. Its foundations video resources for continuous professional development were the result of the expectations of a surgery community has also been evaluated by other authors [8,9]. It is also of- related to the access to a surgery site created by surgeons ten stressed that the use of video on-demand resources al- and targeted at surgeons and their patients. The diffusion of lows us to overcome the barriers of learning [10]. Such bar- surgical knowledge was carried out through the use of vid- riers are a considerable distance between health-care facili- eos and 3D animated computer graphics. ties from main medical educational centres. The continuous Some authors put a particular focus on limitations re- medical education resources may be delivered to health lated to low bandwidth of connections to the digital video professionals working in rural or peripheral medical facili- library. This issue becomes a real problem when we have ties as interactive video courses or in asynchronous mode. in mind the phenomenon of “digital divide”1 occurring in This latter option is particularly important for clinicians many states in the world and hampering, the access of who can review the available material at a convenient mo- health professionals in numerous developing countries. ment. The access to online health databases, including an MDVL, may be an important trace of continuous medical education [6,11]. The value of streaming video was appre- 1 The gap between those people with effective access to ciated by the editors of the Annals of Thoracic Surgery, who digital and information technology, and those without access to it.

Opto-Electron. Rev., 16, no. 4, 2008 M. Duplaga 429 Compression evaluation of surgery video recordings retaining diagnostic credibility

It should be mentioned that a rapid development of tests and imaging techniques. The introduction of flexible wireless networks has opened a possibility for accessing bronchoscopes enhanced the range of tracheobronchial tree MDVLs using mobile terminals, i.e., portable computers accessible to inspections, as well as the scope of sampling (laptops), handheld computers (palmtops) or cellular pho- methods. The bronchoscopy is frequently accompanied by nes. Obviously, as the displays used for watching video se- procedures focused on obtaining the samples eligible for quences do not have enough resolution, several limitations histopathological or microbiological evaluation. Those in- occur when it comes to using the solution for diagnostic clude scissors biopsies of mucosa and lung tissue, trans- purposes [15]. bronchial needle aspiration, brushing and lavages. The The basic advantage of using an MDVL instead of a tra- choice of the appropriate site for sampling is highly impor- ditional VHS tape archive is clearly visible mostly where tant when diagnosis of neoplastic lesions is at stake. As the search and delivery time for a video sequence (e.g., a record- efficiency of bronchoscopy is generally dependent on the ing of a medical operation) is significant. Thanks to an skills and experience of the physician performing the ex- MDVL armoured with indexing features, finding such a amination, the options for obtaining peer-review on seen video sequence and starting its play-out takes no longer than images becomes crucial for feedback on the quality of the seconds. Therefore, an MDVL seems to be a very effective service. In this context, the use of relevant technical means tool along with its features, an important source of digital in- supporting the recording and storage of performed proce- formation, possibility to integrate with other systems, stre- dures enables peer-based assessment of the recorded bron- aming and indexing functionality, video compression, etc. choscopic images. The library of video recordings also The last feature, especially if done by means of lossy codecs, brings considerable educational value for trainees. becomes an important characteristic of every MDVL. Unfortunately, prospects of exploiting (for diagnostic 1.2. Characteristics of bronchoscopic recordings purposes) images (including moving images) with visible lossy compression seem to be difficult to realise, as a serious hazard of impermissible influence on diagnosis results Even if we feel that we are entering a digital era, the use of an- does exist. The approach states that it is possible to use (for alogue video systems is still common in healthcare. The vol- diagnostic purposes) images in which lossy compression ume of resources, which could be used more efficiently due to opportunities provided by digital technology, is tremendous. has not caused any distortion visible to a panel of physi- The use of an MDVL enables enhanced storing, editing and cians (in other words, compressed images and video se- sharing of these resources. The solutions implemented in spe- quences are visually lossless compressed). cific locations may explore various technical options. Therefore, it seems reasonable to assume, according Video sequences containing surgery content and stored to [16,17], that it is possible to use lossy compressed im- in an MDVL, especially if recorded by medical devices, are ages and video sequences for diagnostic purposes pro- often characterized with non-standard features. For example, vided the compression has not introduced any quality im- a motion activity concentrates only in a small area of image. pairment visible to a panel of physicians (referred to as The same happens to recordings of bronchoscopic pro- images and video sequences which are visually lossless cedures that are a specific case of video sequences as well. compressed). Moving images recorded during bronchoscopic procedures usually consist of the main camera area and some sur- 1.3. State-of-the-art methods for surgery video rounding textual information (Fig. 1). The camera image storage and compression can sustain almost motionless for quite long periods of time (the video sequences are generally rather long, often lasting There are several publications presenting research results for several minutes). This altogether predisposes such on storing compressed surgery video sequences. The ef- video sequences as being very susceptible to compression. fects of this research are visible in creation and develop- Bronchoscopy is one of the basic procedures employed ment of standards for storing and compressing surgery in pulmonary diagnostics, apart from pulmonary function video sequences [18].

Fig. 1. Single frames (screenshots) of three example video sequences recorded during execution of bronchoscopic examinations.

430 Opto-Electron. Rev., 16, no. 4, 2008 © 2008 SEP, Warsaw However, it should be stressed that research on surgery used2. For diagnostic purposes, the most popular format is multimedia data compression is usually performed in narrow AVI. The most commonly used video codec is (almost medical specializations. The examples could be oncology [18] lossless) MJPEG, resulting in broadband streams that are (the EUROPATH project), radiology [16,18] (the not suitable for streaming. EURORAD project), ophthalmology [3], paediatrics [4], sur- On the other hand, lossy codecs generally produce de- gery [5,6,9,14,19,20], nursing [7], dentistry [6], internal medi- sirable narrow-band video streams. Unfortunately, as the cine [10,13,21] emergency medicine [11,12,22], and last but authors have mentioned, lossy codecs are hardly ever used not least in bronchoscopy which is addressed in this paper. for a compression of surgery multimedia data if recon- The most important standard among them is the digital structed images are later supposed to support a diagnostic imaging and communication in medicine (DICOM) (stan- process. These codecs are usually designed with the as- dard, Ref. 23). The DICOM standard has become the basic sumption of introducing a significant loss if used for an ef- standard used for the storage of compressed surgery multi- fective compression (the reconstructed image may differ media data. DICOM allows for storing both still images significantly from the original one), as long as a user is still and video sequences [23], although it has not been de- satisfied with the distortions in the reconstructed video se- signed to be used for long-lasting video sequences. Com- quence. In the case of using visually impaired images for pression algorithms to be chosen for trusty storage of diagnostic purposes, a serious danger of an impermissible source data were one of the key aspects during the develop- influence on the diagnosis does exist. ment process of the standard. Currently, most of the codecs Usually, if a presentation of surgery multimedia data assumed in the DICOM standard are lossless ones. On the does not have to support a diagnostic process, but is used other hand, it is important to note that the DICOM standard for educational purposes, for example, popular consumer is continuously under development and it is likely to in- (usually lossy) codecs are used. In the case of still images, clude more lossy compression methods in the future. the JPEG compression standard [24] and the JFIF format Usage of real lossless codecs results in low compression are used [25]. If video sequences are being compressed, ratios (CR s) currently achieved when compressing and stor- popular codecs (see Table 1) like MPEG-1, MPEG-2, ing video sequences (e.g. in the DICOM format). Therefore MPEG-4, RealVideo 9 and others are used [26]. these codecs implicate waste requirements on both an ar- In conclusion, are we doomed to a contradictory choice chive memory and a throughput of a streaming network. between either narrow-band streams without a diagnostic As far as the storage of long bronchoscopic video re- potential of voluminous file storage or transfers being diag- cordings is concerned, the DICOM standard is usually not nostically credible?

Table 1. Selected compression standards of digital video signals. Family of standards + Characteristic features examples Application area Intra-frame: Individual decoding of independent video sequence frames, low coder and decoder complexity, low com- M-JPEG 2000 [27] pression efficiency (often less than 10:1) Widely used; consumer electronics, studio systems, non-linear editing 2G: MPEG-1 [28] Limited selection of macro-block encoding types, basic Huffman encoding only, single motion vector per P-type macro-block; currently quite outdated (MPEG-1 – 1992); most popular MPEG-1 compression scenarios: 250 kbit/s – 1.1 Mbit/s; CBR and VBR supported Outdated, disappearing systems; real-life shots, Video CD recording 3G: MPEG-2 [29,30], Some compression effectiveness improvements and significant extension of functionality, extended encod- H.263+ [31,32], ing mode selection, interlace signals encoding options, extended selection of quantifiers, alternative Huffman MPEG-4 [33] or even arithmetic encoding options; simple video stream scalability in MPEG-2; object-oriented approach in MPEG-4 allowing for movement and scaling of independent video sequence components; most popular MPEG-2 compression scenarios: 3 Mbit/s – 11 Mbit/s; most popular MPEG-4 compression scenarios: 64 kbit/s – 900 kbit/s Widely used (digital TV, DVD, Internet); CBR and VBR supported; HDTV 4G: MPEG-4 Several improvement driving to compression effectiveness introduced AVC/H.264 [34,35] Currently being introduced into consumer electronics

2 In fact, the newest version of the standard introduces the first extensions toward storage of long bronchoscopic recordings.

Opto-Electron. Rev., 16, no. 4, 2008 M. Duplaga 431 Compression evaluation of surgery video recordings retaining diagnostic credibility

2. Subjective and objective evaluation of lossy cordings into two disjunctive subsets (the visually lossless codecs and lossy). Fuzzy results could always be eliminated if pan- els of physicians, rather than individuals, were asked. This section provides details of procedures and experi- As the first step, a subjective evaluation (in other ments performed toward subjective and objective evalua- words, an observational one, with a cooperation of a panel tion of lossy codecs. The results of both evaluation ap- of eight pulmonologists) of the video sequences com- proaches are presented in Sect. 3. pressed (MPEG-4 standard [33]) at various CR s has been MPEG-4 Premises revealed in the previous section determine the performed. In order to obtain CRmax , for which a clini- maximum CR value for different available types of codecs cian cannot distinguish between the original video se- allowing for diagnostically credible compression of long quence and the compressed video sequence, the test based surgery video sequences. Recordings of bronchoscopic on the above-mentioned quality-based ordering method has procedures have been examined as a case study. The exper- been executed. Each of the three original (uncompressed) imental approach has been defined as follows: video sequences has been complemented with a few video l sequences having the same content, compressed using the to specify the CRmax (still yielding in a visually lossless compression) for some selected video codec (a refer- MPEG-4 standard, thus constituting three investigated ence codec) using subjective evaluations supported by a video sequences. The MPEG-4 codec [33] has been se- physician panel, lected as the reference codec as it is a modern, open, and l to specify numerically the corresponding objective dis- still widely used solution. The codec has been successfully tortion metrics for the same video sequence compressed applied for surgery video compression as well [17,38]. The compression has been done with the CR equal to approxi- with the reference codec at the CRmax, l mately 16 (the minimum CR), 32, 64, 96, 128, 256, and to estimate CRmax values for other codecs under consid- eration at the same objective distortion values as for the 512 (the maximum CR)4. reference codec. Eight clinicians (working on bronchoscopic recordings on a daily basis) have been asked independently to order In order to determine the CRmax value of a codec (allowing for a visually undistorted compression), three (se- the video sequences (from the best quality video sequence lected and prepared in cooperation with a pulmonologist) to the worst quality video sequence) in each of the sets fol- recordings of bronchoscopic examinations have been used. lowing the well-known bubble sort algorithm [39,40]. The Each video sequence contained a recording with a different only information gathered was order (by quality) of the disease type (the disease type has not been taken into ac- video sequences. For the purpose of ordering, the special- count in further analysis). ists have used the software developed specially for this purpose (see Fig. 2). The software has been run at a regular 2.1. State-of-the-art methods for surgery video personal computer located at the clinic. No time restrictions storage and compression for evaluations have been applied.

The published [36,37] and tested subjective method for quali- 2.2. Objective evaluation of a reference codec fying elements of a set of lossy compressed still images to the subset of images being visually undistorted is based on order- Unfortunately, most objective methods are designed to ing compressed images by their quality. The same approach evaluate still images. In order to use them for moving pic- has been adopted in the undergoing investigation of video se- tures it seems reasonable to average results over all frames quences. An expert (a tester, being a pulmonologist in this of a given video sequence. Such an approach does not case3) is presented (in a random order) with several video se- touch object motion smoothness. This is a weak restriction quences, the original one and seven others compressed with as investigated recordings of bronchoscopic procedures are various CR values. As the result of ordering, in the prevailing rather motionless. number of executed tests, it is possible for an experiment su- Two objective metrics have been selected for the exper- pervisor to distinguish two subsets of video sequences iments as being prospectively suitable for bronchoscopic [36,37]. The first one consists of the highest quality video se- recordings. The metrics were the hybrid vector measure 5 quences in the random order. The rest of the video sequences (HVM) [36] and the graphical Hosaka plots [41] . appear in the second lowest quality subset (in the random or- The HVM and Hosaka plots are universal metrics since der as well). Only the video sequences belonging to the first one can deploy them to compare images compressed using subset are considered to have the quality which makes them various codecs [16,41]. The HVM metric is optimized with suitable for diagnostic purposes.

The research [36,37] revealed that it is highly unlikely 4 to achieve fuzzy results on segmenting images or video re- In the assumptions, CR s should have been identical for each video sequence. In the practice it is difficult to achieve the exact assumed CR during a compression. 5 Other widely used, but less accurate, video quality metrics 3 The testers have been recruited at the Jagiellonian are: Mean Square Error (MSE), Peak Signal to Noise Ratio University, Collegium Medicum. (PSNR) and Video Quality Metric (VQM).

432 Opto-Electron. Rev., 16, no. 4, 2008 © 2008 SEP, Warsaw Fig. 2. A screenshot of the software developed specially to order the video sequences; the clinicians ordered the video sequences using the bubble sort algorithm [39,40].

MPEG-4 a subjective test and its strong correlation with the diagnos- After specifying CRmax and corresponding HVM tic value of medical images has been proved [16]. The and Hosaka metrics, objective tests for other selected codecs HVM HVM approach produces a vector [,,VKV12 ], each of were repeated. its six coefficients measures some kind of an image depre- In order to determine, for each codec its specific feasi- ciation. The Hosaka metric has been chosen for comparison ble CRmax, a compression ratio was changed as long as purposes as a case study of a non-medical metric that is not HVM (all coefficients) and PH metrics became equal to the optimized for medical images. The Hosaka approach deliv- ones obtained for the MPEG-4 codec. ers the polygon PH whose area is a measure of picture im- pairment. Both metrics have been thoroughly presented in 3. Results of codec evaluation Appendix A. MPEG-4 Once the feasible reference CRmax has been ob- This section presents the results for both the subjective and tained (via subjective tests) objective tests under the objective evaluation approaches. HVM [16] and the Hosaka approach [41] (averaged over all frames of the analysed video sequence) have been 3.1. Subjective evaluation results executed mapping HVM and PH metrics to feasible MPEG-4 CRmax . The results of both evaluation approaches are presented in Sect. 3. For the first set of video sequences, a clear-cut partition on a subset of video sequences visually lossless compressed 2.3. Objective evaluation of other codecs (a compression with the CR not exceeding approximately 96) and a subset of lower quality video sequences have For the last stage of the experiment, eight popular, mod- been identified for evaluations carried out by 6 out of 8 ern video signal codecs (six standardised codecs: physicians. MJPEG 2000 Lossless, MJPEG 2000, MPEG-2, H.263+, For the second and the third set of video sequences, for H.264 Lossless, H.264, and two proprietary codecs: the evaluations of 7 out of 8 clinicians (in each of the sets), Windows Media 9, RealVideo 10) have been chosen. the partition has been observed for a compression with Both lossy and real lossless (available in case of a CR not exceeding approximately 128 (exactly 118 and MPEG-4 AVC/H.2646 and MJPEG2000 video codecs) 124 respectively). compression modes have been selected. In the rest of the evaluations, no clear threshold was observable when the ordering within sets seemed to be random. Most of these random orderings were authored by the same clinician, which may indicate his or her evaluation 6 Referred to later as “H.264”, contrary to “MPEG-4”. airiness. These results have been excluded.

Opto-Electron. Rev., 16, no. 4, 2008 M. Duplaga 433 Compression evaluation of surgery video recordings retaining diagnostic credibility

Table 2. A calculated video sequence distortion rate according to the HVM for a given CR and various codecs. HVM components

Codec CR V1 V2 V3 V4 V5 V6 MPEG-4 (a reference codec) ~96 10.89 291 7.29 5.19 9.05 47.59 MJPEG 2000 Lossless ~2 0.00 0 0.00 0.00 0.00 0.00 MJPEG 2000 ~21 0.36 49 5.32 2.28 0.01 0.11 MPEG-2 ~96 8.46 1019 3.08 5.01 27.89 23.59 H.263+ ~96 1.90 158 8.41 3.98 0.34 1.09 H.264 Lossless ~7 0.00 0 0.00 0.00 0.00 0.00 H.264 ~96 1.75 103 7.22 5.02 0.26 0.89 H.264 ~100 1.76 104 7.26 4.63 0.26 0.90 H.264 ~104 1.77 106 7.33 4.32 0.27 0.95 H.264 ~112 1.79 111 7.43 3.66 0.27 0.95 H.264 ~128 1.83 120 7.61 3.27 0.29 0.99 Windows Media 9 ~96 1.09 131 7.69 5.24 0.09 0.45 RealVideo 10 ~96 10.36 755 6.58 4.56 10.04 26.01

Table 3. A calculated video sequence distortion rate according to 3.2. Objective evaluation results the Hosaka metric for a given CR and various codecs. Table 2 and 3 present results of the objective evaluations of Codec CR P H other specified codecs. In the tables, results meeting the MPEG-4 (a reference codec) ~96 8.18 (computed) condition of visually lossless compression have MJPEG 2000 Lossless ~2 0.00 been indicated with bold font. MJPEG 2000 ~21 0.08 As far as the objective tests are concerned, the simplest task is to compare lossless compression standards as they MPEG-2 ~96 16.30 do not introduce any distortions in compressed video se- H.263+ ~96 0.21 quences. Therefore, the only comparison criterion is the H.263+ ~128 0.37 CR, usually achieving low values. Definitely, the best re- H.263+ ~256 2.57 sults within the family of lossless standards have been achieved using the H.264 lossless standard (CR = 7). H.264 Lossless ~7 0.00 The MJPEG2000 compression standard was the single H.264 ~96 0.08 compression standard without an inter-frame compression. H.264 ~128 0.10 In the lossless version compression is running only with the H.264 ~256 0.24 low CR = 2, however, in the lossy version it is possible to achieve CR = 21 while preserving visually lossless com- H.264 ~512 0.26 pression. Nevertheless, it is still a low CR if compared to Windows Media 9 ~96 0.14 the one achieved for the MPEG-4 standard compression Windows Media 9 ~128 0.31 (CR = 93, and thus CR » 96). Windows Media 9 ~256 0.72 In the family of the third generation standards, the MPEG-2 and H.263+ standards have been tested. Compres- Windows Media 9 ~512 5.95 sion in the MPEG-2 standard with CR » 96 allowed a rela- RealVideo 10 ~96 15.65 tively low amount of error level, except the maximum pixel error V2 and the integral square with frequency weighting MPEG-4 defined by CCIR V . Compression in the H.263+ standard Being aware that the CRmax value obtained using 5 the presented approach might always be over optimistic achieved similar results to MPEG-4, except for errors cor- - related in the5×5window, being relatively large for the one, it has been cautiously decided to set CRMPEG 4 » 96 max H.263+ codec. as the worst obtained result in subjective evaluations. The A possibility for achieving a higher CR while sustaining MPEG-4 » CRmax 96 threshold is associated with the corre- the condition of being visually lossless compressed could sponding HVM and PH metrics as given in Table 2 and 3. be ensured only by the newest, fourth generation of codecs

434 Opto-Electron. Rev., 16, no. 4, 2008 © 2008 SEP, Warsaw (H.264, WM9). For the H.264 compression standard it is It is strongly recommended to use the HVM metric for possible to compress with the CR » 100 keeping all HVM objective evaluation of medical video sequences, although coefficients below their thresholds. Unfortunately, for the the analysis of the results revealed that sometimes the V4 Windows Media 9 codec, compression even for CR » 96 coefficient does not follow the monotonic growth of the CR HVM makes V3 and V4 coefficients exceed their thresholds. as a result of frame averaging. As far as the Hosaka An additional issue, appearing while evaluating video metric is concerned, the subjective, ex-post-visual analysis sequences using still image evaluation methods, is passing of video sequences performed by the authors contradicts over object motion smoothness. The problem appeared the results achieved under the metric, which probably ex- clearly during the RealVideo 10 codec evaluation. The cludes it from video evaluation. codec, contrary to the rest of the tested codecs, does not use The presented research and its results concern video se- a constant frame per second rate (FPS) while compressing quences recorded during bronchoscopic procedures. How- ever, the proposed approach to combine subjective and ob- video sequences. The FPS fluctuates according to objects’ jective evaluations could be applied for most other long- motion activity. In consequence, there is no method for di- lasting surgery video recordings featuring a low level of mo- rect comparison of original and reconstructed video se- tion. This will allow for constructing MDVLs containing quence frames. Therefore, a frame interpolation has been high quality video content recorded during bronchoscopic accepted for objective evaluations. procedures. The MDVLs can be used for diagnostics as well The analysis of the results achieved using the HVM as for documentary and educational purposes. Considering metric revealed that the V4 coefficient does not follow the several similarities between bronchoscopic recordings and monotonic growth of the CR as a result of HVM frame av- other endoscopic recordings (like colonoscopy and gastro- eraging. The authors explain this as an effect of averaging scopy), it is likely that the promising results are to be ex- the coefficients over all frames of an analysed video se- tended into these areas as well. quence. Even if growth of the V4 caused by growth of the is observable for still images (as the HVM operates on still Acknowledgements images only), quality loss for various frames of a video sequence being compressed can proceed with various speeds. The work presented in this paper was supported in part by The comparison of codecs under the Hosaka metric the European Commission under grant FP6-506869 (NoE gave surprising results that, for some codecs compression E-NEXT) and the Polish Ministry of Science and Higher with even the very high CR could allow for visually Education under grants 3 T11D 017 27 and N 517 4388 33. lossless compression. On the other hand, the subjective, The authors thank the panel of physicians from ex-post-visual analysis of these video sequences performed Jagiellonian University, Collegium Medicum for their ef- by the authors contradicts the results achieved under the forts and commitment during subjective evaluations. Hosaka metric, which probably excludes it from video Appendix A. Objective metrics evaluation. 4. Conclusions The hybrid vector measure (HVM) is one of the acknowledged metrics for an objective evaluation of a degree of depreciation introduced by compression. The metric is an ex- Based on bronchoscopic recordings, it has been proved that tension of the idea of picture quality scale (PQS) [42] in the it is possible to effectively compress long-lasting and slow direction of graphical vector metrics (the graphical presen- motion video sequences of surgery procedures while pre- tation method has been created), as well as in the direction serving their diagnostic features. Effective, in terms of of a diagnostic reliability. The HVM is a vector metric. The bandwidth and diagnostically credible archiving and vector has been defined as a vector of six coefficients: streaming of surgery content, is the sine qua non for a rapid l measures of point accuracy errors (PAE): dissemination of medical digital video libraries. – V1, the average pixel error. This factor character- In particular, the authors found out that it is possible to ises a mean point error and hence reconstruction apply the third generation MPEG-4 standard for a compres- accuracy. Because it is the mean difference be- sion of video sequences supporting diagnostic processes tween the values of the original) image fxy(, )and for bronchoscopic imaging data at a highly satisfactory the reconstructed image fxy(, ), as the integral- compression ratio of about 96. The other third generation manner measure, does not capture individual picks codecs did not provide satisfactory results. It has been con- of image quality, but rather shows a general level firmed as well that older generation codecs, as well as of a pixel reconstruction accuracy. lossless codecs, do not allow for compression with the di- – V2, the maximum pixel error. The maximum differ- agnostic quality along with CR » 96. It is possible to ence of corresponding pixel values is an important achieve higher CR (CR » 100) while sustaining the condi- factor for preserving small, diagnostically impor- tion of diagnostic quality only by the newest, fourth gener- tant structures which must not be changed in an ar- ation, H.264 compression standard. chiving process. This differential-manner measure

Opto-Electron. Rev., 16, no. 4, 2008 M. Duplaga 435 Compression evaluation of surgery video recordings retaining diagnostic credibility

is a good supplement of V1, and both are defined in the original and the reconstructed image data domain. l measures of local structured errors (LSE):

– V3, the correlated errors in a 5 × 5 window. This factor characterises a local spatial correlation and it is defined as the summation over the entire image of a local error correlation. The sums are computed over the set of pixels in the5×5window.

– V4, the preservation of high contrast edges. The V4 factor deals with psycho-physical effects which affect the perception of errors in the vicinity of high contrast transitions. It is visual masking which re- fers to the reduced visibility of disturbances in ac- K tivity areas. Fig. 3. A graphic form of the HVM: the six factors (,VV16 , )are l measures of random errors (RE): divided into three groups: a “PAE” informs about point errors (V1 and V2), an “LSE” field represents structured errors (V3 and V4) and – V5, the integral square with a frequency weighting defined by CCIR. This factor is defined similarly to an “RE” rectangle is a sign of random errors (V5 and V6). a normalised mean square error with a frequency 6 HVM = Sa weighting defined by CCIR 567-1. The V5 factor is dW(,0 ) ii. defined in PQS as well. Together with the next fac- i= 1 aaK tor, they characterise the energy of the difference It should be noted that the weights (,,16 )follow K between original and reconstructed images. Ran- the vector coefficients (,VV16 , ). The weights are adjusted dom disturbances introduced by a codec, or random to maximise the correlation between the HVM scalar equi- errors of the original image reduced by a codec, can valent with average values of subjective scores. Weight be accurately described by these factors. sets differ according to a medical data type or even a recording device type, making it difficult to use a homoge- – V6, the integral square normalised by pixel values. This metric gives additional information about ran- neous scalar HVM for various types of video sequences. As dom errors without frequency weighting. a result, the authors have decided not to use the scalar The HVM is originally a vector metric. Growth of each HVM equivalent in research for the objective, universal of the six coefficients caused by even stronger compression evaluation of selected codecs. means depreciation of some image feature (a comparison Another acknowledged method for an objective evalua- of two identical video sequences should zero all the vector tion of a rate of distortion introduced by a compression is a coefficients). usage of graphical Hosaka plots. The Hosaka plots are an An average HVM of processed video sequences contain objectively computational comparison method, allowing to a graphical form of presentation of distortions in order to specify the fidelity of intensity reconstruction as well as an characterise them better and analyse them deeper. Thanks to multicolour rectangles, three groups of errors are visual- ised: point accuracy errors (PAE, V1, V2 ), local structured errors (LSE, V3, V4) and random errors (RE, V5, V6). Inten- sification of distortions causes an enlargement of areas of rectangles, which should correspond with a negative mean- ing of the coefficients defined by these three pairs. An example HVM plot is depicted in Fig. 3. Qualities of two video sequences can be compared with each other using a scalar HVM equivalent, defined on the basis of the so-called Manhattan distance. Distance between two vectors a and b is defined in the metric as fol- ab =-Sa a lows, dab(, )ii | i |, where i are the positive weights. i Assuming that an original video sequence (having the zero vector) is being compared with a video sequence reconstructed after compression, having some HVM, the distance between these two vectors (HVM is the scalar equivalent of HVM ) is calculated as a sum of the six values (vector coefficients) Fig. 4. An example of a weighted Hosaka plot.

436 Opto-Electron. Rev., 16, no. 4, 2008 © 2008 SEP, Warsaw additive noise, both in pixel blocks divided into a few 16. A. Przelaskowski, Wavelet-based Image Data Compression groups – from 1×1to16×16.Thearea of plots in the left (Falkowe metody kompresji danych obrazowych), Publi- half (called the noise area) is more sensitive to an additive shing House of the Warsaw University of Technology noise, while the right side area (called the fidelity of recon- (Oficyna Wydawnicza PW), pp. 149–190, Warsaw, 2002. struction area) reflects more precisely the fidelity of inten- 17. C. Dafonte, A. Gomez, A. Castro, and B. Arcay, “Scientific sity reconstruction, e.g. blurring effects introduced by a papers: a proposal for improving ICU assistance through in- telligent monitoring and supervision”, Technol. Health. compression method (see Fig. 4.) [36,37]. Care 10, 464–466, 2002. References 18. Rubis Project, Healthcare Telematics Projects, Rubis Pro- ject Final Report, pp. 17 – 19, Rubis Project, 2001. 19. B. Malassagne, D. Mutter, J. Leroy, M. Smith, L. Soler, and 1. S.T.C. Wong, and D. Tjandra, “A digital library for biomed- J. Marescaux, “Teleeducation in surgery: European Institute ical imaging on the Internet”, IEEE Commun. Mag. 37, for Telesurgery Experience”, World J. Surg. 25, 1490–1494 84–91 (1999). (2001). 2. R. Tadeusiewicz and M.R. Ogiela, Medical image under- 20. J. Rosser, B. Herman, and C. Ehrenwerth, “An overview of standing technology. Artificial intelligence and soft-com- video streaming on the Internet and its application to surgi- puting for image understanding, Springer-Verlag, Berlin, cal education”, Surg. Endosc. 15, 624–629 (2001). 2004. 21. S.A. Zollo, M.G. Kienzle, Z. Henshaw, L.G. Crist, and D.S. 3. H. Miron, and E.Z. Blumenthal, “Bridging analog and digi- Wakefield, “Tele-education in a telemedicine environment: tal video in the surgical setting”, J. Cataract. Refr. Surg. 29, Implications for rural health care and academic medical 1874–1877 (2003). centres”, J. Med. Syst. 23, 107–122 (1999). 4. N.M. Hamilton, I. Frade, P. Duguid, J. Furnace, and A.D. 22. M.A. Gisondi, S.V. Mahadevan, S.S. Sovndal, and G.H. Kindley, “Digital video for networked CAL delivery”, J. Gilbert, “19 Emergency department orientation utilizing Audiov. Media. Med. 18, 59–63 (1995). Web-based streaming video”, Acad. Emerg. Med. 10, 920 5. S.A. Kumar, and H. Pal, “Digital video recording of cardiac (2003). surgical procedures”, Ann. Thorac. Surg. 77, 1063–1065 23. National Electrical Manufacturers Association, “National (2004). Electrical Manufacturers Association. Digital Imaging and 6. P.A. Reynolds, and R. Mason, “On-line video media for Communications in Medicine (DICOM)”, in Digital Imaging continuing professional development in dentistry”, Comput. and Communications in Medicine (DICOM), National Electri- Educ. 39, 65–98 (2002). cal Manufacturers Association, http://medical.nema.org, 2004. 7. S.M. Green, D. Voegeli, M. Harrison, J. Phillips, J. Know- 24. International Organization for Standardization, ISO/IEC les, M. Weaver, and K. Shephard, “Evaluating the use of 2022:1994: Information technology – Character code struc- streaming video to support student learning in a first-year ture and extension techniques, pp. 47, International Organi- life sciences course for student nurses”, Nurs. Educ. Today zation for Standardization, Geneva, 1994. 23, 255–261 (2003). 25. H.J. Lowe, “The new telemedicine paradigm: Using Inter- 8. C.D. Wright, Digital Library Technology Trends, Sun net-based multimedia electronic medical record systems to Microsystems Inc., 2002. support wide-area clinical care delivery”, in Telemedicine 9. P.S. Greene, “Streaming video for the annals Internet read- and Telecommunications: Options for the New Century, ers”, Ann. Thorac. Surg. 77, 1063–1065 (2004). Bethesda, 2001. 10. J. Strom, “Overcoming barriers for teaching and learning”, 26. M. Cuggia, F. Mougin, and P. Le Beux, “Indexing method in Proc. Int. Symp. Educational Conferencing, Banff, 2002. of digital audiovisual medical resources with semantic Web 11. R.M. Lavitan, T.S. Goldman, D.A. Bryan, F. Shofer, and A. integration”, Int. J. Med. Inform. 74, 169–177 (2005). Harlich, “Training with video imaging improves the initial 27. International Organization for Standardization, ISO/IEC intubation success rates of paramedic trainees in an operat- 15444-3:2003: Information technology – JPEG 2000 image ing room setting”, Ann. Emerg. Med. 37, 46–50 (2001). coding system – Part 3: Motion JPEG 2000, International 12. J. Leung, G. D’Onofrio, B. Duncan, R. Trepp, N. Vasques, Organization for Standardization, Geneva, 2003. and J. Schriver, “Apply streaming audio and video technol- 28. ISO/IEC, “MPEG-1 Coding of moving pictures and associ- ogy to enhance emergency physician education”, Acad. ated audio for digital storage media at up to about 1.5 Emerg. Med. 9, 1059 (2002). Mbit/s”, ISO/IEC 11172 (1993). 13. J.M. Wiecha, R. Gramling, P. Joachim, and H. Vander- 29. ISO/IEC, “Information technology – Generic coding of schmidt, “Collaborative e-learning using streaming video moving pictures and associated audio information: Video”, and asynchronous discussion boards to teach the cognitive ISO/IEC 13818-2 (2002). foundation of medical interviewing: A case study”, J. Med. 30. Ii Recommendation Itu-T, Recommendation ITU-T H.262 Internet Res. 5, 13 (2003). (1995 E) i, 1995. 14. A. Gandsas, K. McIntire, G. Palli, and A. Park, “Live 31. ITU-R, “Video coding for low bit rate communication”, streaming video for medical education: a laboratory model”, Recommendation ITU-R H.263, 01 (2005). J. Laparoendosc. Adv. A12, 377–382 (2002). 32. W.K.H. Ho, W.K. Cheuk, and D.P.K. Lun, “Content-based 15. M. Leszczuk, “Accessing digital video libraries from mo- scalable H.263 video coding for road traffic monitoring”, bile terminals in 3G networks”, in Proc. Advanced Technol- IEEE T. Multimedia 7, 615–623, 2005. ogies, Applications and Market Strategies for 3G 33. ISO/IEC, “Information technology – coding of audio-visual ATAMS’2001, pp. 164–171, Kraków, 2001. objects – Part 2: Visual”, ISO/IEC 14496-2, 6 (2004).

Opto-Electron. Rev., 16, no. 4, 2008 M. Duplaga 437 Compression evaluation of surgery video recordings retaining diagnostic credibility

34. International Organization for Standardization, ISO/IEC pressed DICOM video data”, in Connecting Medical Infor- 14496-10:2004: Information technology – Coding of au- matics and Bio-Informatics, Section 7: Imaging Informatics, dio-visual objects – Part 10: Advanced Video Coding, Inter- pp. 447–452, edited by R. Engelbrecht, ENMI, 2005. national Organization for Standardization, Geneva, 2004. 39. D.E. Knuth, The Art of Computer Programming Volumes 35. ITU-R, “Advanced video coding for generic audiovisual 1-3 Boxed Set, Addison-Wesley Longman Publishing Co., services”, Recommendation ITU-R H.264, 03 (2005). Inc., Boston, 1998. 36. A. Przelaskowski, “Vector quality measure of lossy com- 40. T.H. Cormen, C. Stein, R.L. Rivest, and Ch.E. Leiserson, pressed medical images”, Comput. Biol. Med. 34, 193–207 Introduction to Algorithms, McGraw-Hill Higher Education, (2004). 2001. 37. Multimedia – Compression Algorithms and Standards,ed- 41. K. Hosaka, “A new picture quality evaluation method”, in ited by W. Skarbek, Akademicka Oficyna Wydawnicza Proc. Int. Picture Coding Symp., pp. 17–18, Tokyo, 1986. PLJ, 1998. (in Polish) 42. M. Miyahara, K. Kotani, and V.R. Algazi, “Objective pic- 38. T. Frankewitsch, S. Soehnlein, M. Mueller, and H.U. ture quality scale (PQS) for image coding”, IEEE T. 46 Prokosch, “Computed quality assessment of MPEG4-com- Commun. , 1215–1226 (1998).