Programming Language Support for Digitized Images Or, the Monsters in the Closet
Total Page:16
File Type:pdf, Size:1020Kb
The following paper was originally published in the Proceedings of the Conference on Domain-Specific Languages Santa Barbara, California, October 1997 Programming Language Support for Digitized Images or, The Monsters in the Closet Daniel E. Stevenson and Margaret M. Fleck University of Iowa For more information about USENIX Association contact: 1. Phone: 510 528-8649 2. FAX: 510 548-5738 3. Email: [email protected] 4. WWW URL:http://www.usenix.org Programming Language Supp ort for Digitized Images or, The Monsters in the Closet y Daniel E. Stevenson Margaret M. Fleck Department of Computer Science Departmentof Computer Science University of Iowa University of Iowa Iowa City,IA 52242, USA Iowa City, IA 52242, USA Abstract rapid spread of images is particularly obvious on the World-Wide Web. Although many users require only image pro cessing Computer vision image understanding algorithms and graphics packages e.g. xv, Photoshop, an in- are dicult to write, debug, maintain, and share. creasing range of applications require basic image This complicates col laboration, teaching, and repli- understanding. For example, researchers in the sci- cation of research results. This paper shows how ences and medicine use images to measure physical user-level code can be simpli ed by providing bet- prop erties e.g. blood vessel width and screen for ter programming language constructs, particularly interesting events e.g. unusual cell shap es. Com- a new abstract data type cal led a \sheet." These panies, governments, non-pro t organizations e.g. primitives have been implemented as an extension museums, and private citizens are converting photo to Scheme. collections to digitized form. They require e ective Implementation of sheet operations is made chal- ways to lo cate images with sp eci c content in large lenging by the fact that images are extremely large, databases. e.g. sometimes over 5 megabytes each. Therefore, Computer vision lies on the b order between com- operations that loop through images must be com- puter science and electrical engineering. Tradition- piled from a specialized subset of Scheme into C. ally, it has b een isolated from the rest of computer This paper discusses how the need for extreme e- science. In particular, it has seen little b ene t from ciency a ects the design of the user-level language, recent advances in the design and implementation the run-time support, and the compiler. of programming languages. Computer vision algo- rithms are still written primarily in C, o ccasionally now in C++. 1 Intro duction As a result, computer vision co de tends to b e com- plex and rep etitive. It is dicult to write, debug, maintain, and share. Inability to replicate results re- Within the past few years, computer imaging equip- p orted in research pap ers, which is the norm rather ment has b ecome dramatically cheap er and more re- than the exception, is a ma jor barrier to progress liable. Six years ago, a color camera and framegrab- in this sub eld. Collab oration with researchers in ber cost $15,000 and was to o heavy to lift. It can other areas of computer science is almost imp ossi- now be replaced by a $400 hand-held digital cam- ble. era. As a result, cameras are b ecoming widely avail- able. Both programmers and researchers are start- More signi cantly, many computer vision algo- ing to incorp orate images into computer science ap- rithms are not much easier to program in high-level plications remote from the traditional home of im- languages in practice, always Common Lisp. O - ages, computer vision image understanding. The the-shelf compilers and interpreters do not provide the required eciency. And the high-level language Now at the University of Wisconsin{Eau Claire, Eau co de tends to resemble a word-for-word translation Claire, WI 54702 y Now at Harvey Mudd College, 301 E. Twelfth St., Clare- of the C co de. Existing abstract data typ es and con- mont, CA 91711 trol constructs do not match the rep etitive structure ically stored as packed bytes, b oth in memory and in this co de and, therefore, cannot b e used to sim- in disk les, even though they are conceptually real plify it. numb ers. Allo cation and deallo cation must b e un- der user control, b ecause images do not naturally This pap er will present new abstract data typ es b ecome garbage in the sense of b ecoming inacces- and asso ciated op erations. These primitives, imple- sible to the user quickly enough to prevent mem- mented as an extension to Scheme [3], encapsulate ory from lling up. They must b e allo cated outside much of the rep etitivework common in computer vi- the heap in a language suchasScheme, to prevent sion co de and can b e compiled into ecient C co de. heap fragmentation and unnecessary copying of im- We will discuss key issues involved in implementing age data. And it must b e p ossible for related images the required compiler and run-time supp ort, com- e.g. an image and a subimage to share storage. plementing previous work on compiling Scheme into C. And we will suggest how some of the features Finally, algorithm designers must b e extremely care- required in computer vision might nd wider appli- ful ab out how image data moves within the com- cation in programming language design. puter. Scanning through an image in the wrong or- der, on a machine with insucient RAM, can cause dramatic swapping. Op erations such as image rota- tion, which must access their input and output in di erent orders, sometimes require that very large 2 Images are huge! images b e divided into subblo cks. Even if there is enough RAM, swapping could still o ccur inside in- The most distinctive feature of digitized images is ternal memory caches. their size. Digital cameras sold for the home PC When image pro cessing or image-related graphics market deliver 24-bit color images at sizes ranging is done on an external b oard, transmitting image from 320 by 240 up to 1600 by 1200. When the data between the b oard and main memory is of- data is stored in packed binary arrays, this trans- ten a signi cant fraction of total running time. Bus lates into between 0.23 and 5.76 megabytes per sp eed is one of the ma jor obstacles in using PC- image. Images obtained from scanners or certain based cameras. On ne-grained parallel pro cessors sp ecialized cameras are even larger, as are video im- e.g. the Connection Machine, transmission of the age sequences and 3D images from medical scanners image from the camera to the pro cessors may com- e.g. MRI. Although they can b e compressed when pletely dominate running time. stored in les, images must remain uncompressed during analysis. Said another way, images are ab out 4-6 orders of magnitude larger than most ob jects traditionally 3 Existing vision packages found in a high-level language, and a single im- age may be comparable in size to the entire heap of a traditional Scheme session. Moreover, a typi- A number of packages of standard image pro cess- cal vision function manipulates several such images ing data structures and op erations have b een devel- e.g. two inputs and one output. Atypical interac- op ed to aid research and teaching in computer vi- tive user doing exp eriments will use up all available sion. Some packages, such as the standard utilities memory in an attempt to manipulate e.g. compare xv and pbmplus, supp ort only simple image pro cess- as many images as p ossible simultaneously. ing and manipulation. Others, such as the HIPS Im- age Pro cessing System [12], Khoros [10], Lab oImage Because images are so large and many applications [6], and Matlab [19], cover a full range of low-level require fast resp onse e.g. tracking moving ob jects, image pro cessing utilities. Finally, some packages computer vision programmers are obsessed with ef- o er supp ort for higher-level vision op erations: the ciency. Only the most stripp ed-down algorithms Image Understanding Environment IUE [11], KB- are fast when iterated over all one million lo cations Vision [7], Vista [14], The Iowa Vision System [4], in an image. Hand-co ding critical functions in as- OBVIUS [5], and the Radius Common Development sembler has only recently b ecome rare. This is a Environment [17]. Many large sites have at least one harsh environment in which to test programming in-house package. And additional packages are used language metho ds. to supp ort image understanding pro jects in scien- ti c elds e.g. astronomy, biology, medicine. Toavoid wasting scarce space, image values are typ- A typical computer vision package contains a li- b er of high-level op erations in each package is quite brary of C or C++ image op erations and an inter- limited. They typically include only one mo dern preted front-end language. The front-end language edge nder typically Canny's, one camera calibra- is used to connect op erations together, to commu- tion algorithm invariably Tsai's, and a small se- nicate with the user, and to implement high-level lection of vision algorithms edge segmentation and \smart" parts of algorithms such as automatic pa- grouping, texture features, motion analysis, classi - rameter setting and control of multi-stage or search cation. algorithms. 3.3 How have they fared? 3.1 Front-end language Although these packages are all well-intentioned, Existing packages use a variety of front-end lan- and incorp orate manyinteresting design ideas, there guages, whose main common feature is that they are is no real prosp ect that any of them will b ecome a always interpreted.