SOCIETY

(L–R) Michael Tompsett, , Nobukazu Teranishi and George Smith (not pictured) received the Queen Elizabeth Prize for Engineering for three innovations that have radically changed the visual world © Jason Alden IMAGE REVOLUTIONARIES In February 2017, the four engineers responsible for the creation of sensors were awarded the Queen Elizabeth Prize for Engineering. Science writer and broadcaster Geoff Watts spoke to them about the development of the technology and how their contributions have revolutionised the way that visual information is captured and analysed.

INGENIA ISSUE 73 DECEMBER 2017 23 IMAGE REVOLUTIONARIES SOCIETY

The market for image-sensing technology is worth some $10 billion EXPLORING THE UNIVERSE annually and could be half as large again by 2020 Euclid is a European Space Agency mission due to be launched in 2020. Its ambitious aim is to investigate the dark energy that accounts for three In selecting the winners of the (PPD); and Eric Fossum, who Clarke CBE. He was teasing us, electronically on the surface of a quarters of the energy/matter 2017 Queen Elizabeth Prize developed the complementary but in this context, his wit has CCD, could be used to store and content of the universe, and for Engineering (QEPrize), metal-oxide semiconductor a double relevance. While most transfer information of the kind may explain why the rate the judges have managed to (CMOS) sensor. These users of traditional film cameras required by computer memories. © European Space Agency expansion of the universe is to more basic physics and, when ‘p’ side contains an excess of encompass four key stages in interlocking developments had at least a dim idea of how In fact, the CCD was soon to find accelerating. he joined the NEC Corporation holes. Teranishi added a p layer the history of solid-state image have had an impact to match their instruments captured light, its major use in a quite different Crucial to the success of the mission are Euclid’s reliable, in 1978, to image sensing. with a high level of impurities sensing that are embodied in the extent of their begetters’ this is much less true of the sphere, but the importance of lightweight, high-performance image sensors for observing the Photodiodes of that period were on top of the n layer of a normal four exceptional individuals. ingenuity. Image sensing has billions who reap the benefits Boyle and Smith’s invention was visible and near-infrared output of the billions of faint galaxies plagued by deficiencies such as photodiode. This modification They are George Smith, who proved to be a technology as of digital imaging every day; self-evident, and celebrated in towards which they will be directed. The focal plane of the visible noise and image lag, so Teranishi had the effect of eliminating invented the charge-coupled socially influential as it is fast and the images themselves, 2009 when the pair received the light instrument will comprise an array of 36 CCDs (as seen above) set out to unravel the roots of image lag, reducing dark current device (CCD); Michael Tompsett, moving. whether a picture of some far Nobel Prize for Physics. designed for the exceptional circumstances in which they will be these problems and looked for (the current generated even who exploited its potential “Any sufficiently advanced distant galaxy or the interior of The real value of the CCD operating. They are highly efficient, should be able to tolerate the a means of dealing with them. in the absence of light) and in image sensing; Nobukazu technology is indistinguishable the human gut, often have an began to emerge with its change radiation to which they will be exposed, and by generating 25 times Two years after joining NEC, he producing a much better image Teranishi, who came up with from magic,” wrote science- emotional resonance that surely of role from memory storage as many pixel images as a top , should offer a far better invented the PPD. quality. It allowed pixel sizes to the modern pinned photodiode fiction novelist Sir Arthur C merits the term ‘magical’? to image sensing, which is quality image. Nothing less would allow astrophysicists to probe the Conventional photodiodes be reduced and their numbers attributable to the second of the history and nature of the universe. comprise a p–n (positive– increased, with a consequent THE CAMERA IN A PILL 2017 QEPrize winners, Michael negative) junction at the boost to resolution. Mass One of the great virtues of the new sensor technology lies THE FIRST Tompsett. Born in the UK, he much smaller than those in use array to one corner of the chip. boundary between two types production started in 1987, and in miniaturisation: in this instance, a camera small enough to DIGITAL PHOTOS studied physics at the University at the time. Here, Tompsett added to the of the same semiconductor a few years later, CMOS image swallow. Capsule endoscopy, as it is called, relies on a camera, an The story behind image-sensing of Cambridge and followed it up Image capturing by CCDs is original CCD design with an crystal. The ‘n’ side contains an sensors could also benefit from LED and a transmitter built into a pill of the size commonly used technology begins with George with a PhD in engineering. By the a product of their response to analogue-to-digital converter excess of electrons, while the the use of PPDs. to deliver drugs or vitamins. As it moves through the stomach Smith. In 1959, after graduating time he came to work on CCDs, light; when photons impinge to turn the amount of charge and the rest of the gut, its camera records images that can be from the University of Chicago he had already demonstrated his on the light-sensitive surface generated within each pixel into transmitted to a recording device worn by the patient. with a PhD in physics, he joined inventiveness. While employed of an array of pixels on a silicon a digital value to be memorised. Conventional endoscopy relies on a long, flexible viewing a department of Bell Telephone by the then English Electric chip, they trigger the formation The digital readout so compiled tube inserted through the mouth and as far down the digestive Laboratories headed by Valve Company, he devised of electrons. These are captured forms a numerical description tract as it will reach. Capsule endoscopy avoids the inevitable Canadian physicist Willard Boyle the uncooled thermal-imaging by the electrostatic forces in of the image. “I led the group discomfort of this procedure, and dispenses with the need for (deceased). In line with the topic television camera tube. However, the pixel and accumulate. The that developed CCD images, and trained staff to perform the examination. of his PhD thesis, he continued by then the focus of his interests brighter the light, the greater the made the first colour camera,” A capsule camera may be tethered on a fine control wire to study semimetals such as had already begun to change. number of electrons generated Tompsett says. “And I took the and retrieved by pulling it back up after examination has been arsenic, antimony and bismuth. “Ideas were being floated within any one pixel site. In an first colour pictures,” he adds. completed. However, as manufacturing costs fall, it can also be He subsequently moved to about solid-state imagers,” he image brought to focus on the “The first person to have a digital disposable and allowed to another department at Bell titled recalls, “and that was something surface of a CCD, the location image taken of her was my wife.” pass out of the body. Device Concepts. In 1969, it was I wanted to work on.” In 1969, and number of electrons form This is still a novel here that he and Boyle invented his fascination took him to the an analogue representation of technology, and the extent the CCD. best place to be for anyone with that image. IMPROVED QUALITY of its benefits in searching Boyle and Smith’s intention such an interest: Bell Telephone Besides acting as a AND SMALLER for cancers or other was to exploit the capacity Laboratories. He quickly realised photodetector, the pixel Tompsett had followed his abnormalities of the digestive of semiconducting materials how Boyle and Smith’s CCDs structure of a CCD also serves father into engineering, and tract is still being assessed, to transfer charge along their could be exploited for their as a readout system. Charge Nobukazu Teranishi also had a generated in the pixels is father who worked in the field. but findings so far are surface from one storage site capacity to capture images, and Michael Tompsett holds a colour camera that he helped to develop while © GivenImaging encouraging. to the next. They reasoned that embarked on a quest to develop transported, step-by-step, from It was nuclear fusion that first his wife Margaret holds a photo of herself, which was the first colour patterns of charge, created imaging systems that were one site to the next, across the attracted him, but that gave way image using three solid-state imagers © Jason Alden

24 INGENIA INGENIA ISSUE 73 DECEMBER 2017 25 IMAGE REVOLUTIONARIES SOCIETY

HANDS-FREE IMAGING TECHNOLOGY RAISING ENGINEERING’S PROFILE In 2010, Microsoft released its motion-sensing device, Kinect. Kinect was an instant hit, selling more than eight million units in its first two months on sale and becoming the then fastest-selling consumer Teranishi moved to the GLOBAL IMPACT sensitivity. The ubiquity of electronics device in history. Panasonic Corporation in 2000, Returning to the present, sensing devices will become Combining depth sensing and subsequently to his present upwards of 50 CMOS cameras even greater as we surround and sensors with academic posts at the University are now being manufactured ourselves with all sorts of machine learning, the gadget of Hyogo and at Shizuoka every second. The market for new and digitally controlled (Left): The three winners of the 2017 QEPrize take a selfie at the University. Here he works on announcement in February 2017 © Jason Alden (Right): Fourth winner, dropped gamers right into the image-sensing technology machines and instruments that George Smith action by turning their bodies X-ray image sensors and photon is worth some $10 billion need to sense and respond to into controllers. counting sensors. annually and could be half their environment. The Queen Elizabeth Prize for Engineering (QEPrize) is a global At the time of its launch, Eric Fossum too has as large again by 2020. While Looking back, to what £1 million prize celebrating the world’s greatest living engineers. Kinect’s learning technology moved between industry and nine out of every 10 sensors extent did the inventive minds Winners are selected by an international panel of judges and are was so advanced that it academia. His introduction sold are now CMOS, CCDs are of this year’s QEPrize winners responsible for a groundbreaking innovation that has benefitted earned the engineers behind to image-sensing technology still preferred when the key anticipate the impact of the humanity worldwide. it the 2011 MacRobert came while he was still in requirements are for higher technologies they had created? Alongside awarding the biennial prize, the QEPrize raises the Award, marking it as the ‘next graduate school, shortly quality and greater light Pointing out that most families public profile of engineering and inspires the next generation big thing’ in engineering. before CCDs began to find to take up the challenges of the future. Its Global Engineering Although Microsoft a widespread application in Ambassadors’ Network, a community of young, diverse role announced in 2017 that it was consumer products. Having CROWDSOURCED DISEASE DIAGNOSES models from across the profession, promotes engineering Small, cheap and universally available, digital image sensors have ending production of Kinect, worked on them at Columbia excellence, while interactive initiatives such as the Create the CMOS sensors paved the way for smaller cameras and the development of turned smartphones into the ultimate diagnostic tool. PlantVillage its groundbreaking technology University in New York, he was Trophy competition invite young people to get involved and the ‘camera on a chip’. Although not as high quality as CCD sensors, they is a free, open-access platform that connects crop farmers with a lives on. recruited by NASA in 1990 and use less power and are cheaper to produce. They are also integrated into experience engineering for themselves. Bringing their ideas to fingerprint recognition on smartphones and are used for identification in global community of experts, diagnosing plant disease from the HoloLens, Microsoft’s moved to its Jet Propulsion life in a mobile and smartphone app, 14 to 24 year olds across biometric passport booths for border control other side of the globe. ‘mixed reality’ headset, Laboratory (JPL) in California. the world competed to design the 3D-printed trophy that was Farmers use their phones to take a photo as soon as they incorporates depth-sensing By this time, CCD devices had it could be implemented) by Despite their advantages, early presented to the QEPrize winners at Buckingham Palace. notice a change in their crops, uploading the image directly technology and the latest proved successful in both British engineer Peter Noble interest in the new CMOS sensors to an online database. With a worldwide network of scientists in digital imaging sensors their consumer and scientific working at Plessey. CMOS was, at best, lukewarm. Fossum had film cameras, Teranishi was to take the picture, and where logging in, they can have an answer to their problem within to capture the wearer’s real- applications, but NASA wanted sensors have most of the left JPL in 1996 having already set always confident that these you were going to take if from, hours. The rapid response allows farmers to quickly control the world environment. The smaller and less power-hungry circuitry and components that up his own company, Photobit, would largely disappear. “What and so on, and then you would disease to save not only their own harvest, but others nearby. device then places holograms CCDs that would also be more they require integrated into the to commercialise CMOS image- we really did not imagine was take just one picture.” Now, of The online platform was created by two assistant professors directly onto physical objects robust in a space environment. sensor itself (hence the term sensor technology. He has since the invention of mobile phone course, we take them by tens at Penn State University’s College of Agricultural Sciences, and is within their field of vision to Fossum’s remedy was to ‘camera on a chip’) and get by returned to academic life and is cameras,” he admits. It is these, or even hundreds, scarcely getting smarter; it learns from an ever-growing image bank and can enhance reality. use CMOS microelectronics on as little as a hundredth of now a professor at Dartmouth of course, that have accounted thinking about it. offer computerised diagnosis. technology in developing a new the power needed by CCDs. College’s Thayer School of for the explosive growth in the As for Fossum, he can While still in its infancy, generation of sensors. While the earliest CMOS devices Engineering in New Hampshire. use of image-sensing devices. still barely believe what has the artificial intelligence Instead of transferring electric performed relatively poorly However, he is still engaged in Not without a hint of happened. As he says, “At matches healthy and charges across the array of pixels by comparison with CCDs, entrepreneurial research; he and nostalgia, Tompsett points to the time [of developing the diseased leaves, and can and reading them at one corner, they have been catching up a former student have a created the near total lack of effort or technology] I was thinking you so far detect 26 different CMOS sensors use transistors ever since. CMOS devices are a new startup company called personal investment in the use could use sensors for this and diseases in 14 plants with within each pixel to amplify the also cheaper to manufacture, Gigajot. “We are working on a modern imaging. “When you sensors for that, and maybe 99% accuracy. Ultimately, voltage signal created there. and they eventually became possible next generation image- took pictures with a film camera they could become ubiquitous, farmers could receive a As Fossum is at pains to point the natural choice for small sensing technology where we you were careful,” he says. but I did not really believe it diagnosis within seconds of out, this is an idea that was consumer electronic devices count each photon of light, one “You were economical about it, would happen.” However, it did © Microsoft HoloLens uploading images. © Pixabay.com/Hans first suggested in 1967 (before such as mobile phones. at a time,” he says. deciding how you were going “and it is truly astonishing”.

26 INGENIA INGENIA ISSUE 73 DECEMBER 2017 27