The Stethoscope Gets Smart

Engineers from Johns Hopkins are giving the humble stethoscope an AI upgrade

By Mounya Elhilali & James E. West

YOU WAKE UP ONE MORNING mend that you travel to a hospital in the nearest to discover that your child is ill: city? Or will she dismiss the symptoms as signs His forehead feels hot to the touch, of a common cold, and send you back home with and his rapid breathing has a your son? If she mistakenly sends you home, will wheezing sound. You live in your son survive the mistake? Malawi, where your health care options are few. Worldwide, more kids die of pneumonia and When the local clinic opens, you wait for your other lung ailments than from any other cause. turn with the solitary clinic worker. She’s not a These acute lower respiratory infections kill nearly doctor, but she’s been trained to identify and 1 million children each year worldwide, causing handle routine problems. more deaths than HIV and malaria combined. But She puts on a stethoscope and presses its chest fewer than 5 percent of people in the developing piece against your son’s front and back to carefully world have access to the X-ray imaging that’s con- listen to his lungs. Through the windows, open sidered the gold standard for pneumonia diagno- in the heat of the day, come the sounds of people ses. What’s more, many sick children never get to talking, the thrum of a generator, and the roar a clinic because they live far away from a facility, of a moped on the main road. The health worker roads are in poor condition, and transportation strains to hear. Will she detect the signs of bacte- is expensive. In sub-Saharan Africa, where most rial pneumonia, give you antibiotics, and recom- pneumonia deaths occur, only two in fve children DR. ERIC D. MCCOLLUM D. ERIC DR.

36 | FEB 2019 | NORTH AMERICAN | SPECTRUM.IEEE.ORG AI ASSISTANCE: A health worker in Bangladesh listens to sounds from a boy’s lungs with the help of the Johns Hopkins smart stethoscope, which cancels out extraneous noise and identifies telltale signs of pneumonia.

SPECTRUM.IEEE.ORG | NORTH AMERICAN | FEB 2019 | 37 Pneumonia with pneumonia symptoms receive pro- GLOBAL KILLER: fessional medical care. In 2015, about 5.9 million children Other Given the global burden of these respi- died before the age 15.5% HIV 1.4% ratory ailments, the World Health Orga- of 5. Pneumonia 24.9% nization has developed strategies for accounted for Malaria 15.5 percent of those countries with limited resources. The deaths around the 5.2% guidelines for pneumonia diagnosis world, and was the minimize dependence on any tech- top cause of death in 8.9% sub-Saharan Africa. 6.1% Diarrhea nological tools, and instead rely solely Injury on observed symptoms of shortness 8.6% of breath, cough, and rapid breathing. In hopes of saving lives, the WHO rec- Congenital abnormalities 29.4% ommends antibiotic treatment for all children with these symptoms, with Preterm and birth the result that half the children who get complications treated for pneumonia don’t really have SOURCE: THE LANCET, VOLUME 388, ISSUE 10063, DECEMBER 2016 it. This approach puts unnecessary costs on communities, and the unnecessary medication contributes to the growing thereby take advantage of its low cost UR MISSION SPRANG problem of antibiotic-resistant bacteria. and ease of use. With funding from the from an ambitious global We propose a technological solution National Heart, Lung, and Blood Insti- study of pneumonia that that builds on the stethoscope, which tute, NASA, and the Bill & Melinda Gates began in 2008, with has hardly changed since its invention in Foundation, we set out to completely funding from the Gates the early 1800s. This ubiquitous clinical reengineer the stethoscope. OFoundation. The Pneumonia Etiology tool has a number of limitations. For best The project was a collaboration among Research for Child Health (PERCH) results, the user should be in a quiet envi- engineers, doctors, and public health study explored the causes of pediatric ronment, as background noise can easily experts at Johns Hopkins University, pneumonia across multiple countries in mask the subtle sounds coming from the in Baltimore. Together, we invented a Africa, Asia, and South America. lungs. The diagnostician must be well device that uses digital sensing tech- In the course of this study, a team of trained in positioning the chest piece nology for sound capture, active acous- doctors from Johns Hopkins set out to properly on the body and in interpreting tics for noise cancellation, and artifcial determine just how useful lung sounds the sounds, which is something of an art intelligence (AI) to help health workers could be in diagnosing pneumonia. At form. Learning to do it well requires time make accurate pneumonia diagnoses. first, the researchers equipped local and mentorship, and even experts can We hope that our smart stethoscope will health workers with commercially disagree on what they hear. That’s why be deployed around the world to prevent available stethoscopes that they had even leading doctors prefer to comple- children from dying of pneumonia. And augmented with small digital record- ment their use of the stethoscope with as we test its capabilities, we may fnd ers. The health workers recorded lung other techniques, such as chest X-rays. that it’s a boon not only for the devel- sounds from the children they encoun- We wondered whether we could over- oping world but also for hospitals in the tered in rural clinics and urban hospitals, come the stethoscope’s limitations and United States and other well-of countries. then transmitted the fles to expert pul- monologists at Johns Hopkins, who lis- tened to these sounds and ofered more

SOUNDS OF TROUBLE: These two spectrograms show digitized audio signals from a informed medical opinions. child who’s breathing normally [left] and from a child who’s producing the high-pitched Quickly, the use of the commercial wheezing noises [right] that indicate fluid and inflammation in the lungs. stethoscope became problematic. To

Normal breathing Breathing with wheezing Hertz Hertz

800 800 20 600 600 400 400 200 200 100 0 1 2 0 1 2

Seconds Seconds UNIVERSITY HOPKINS JOHNS

38 | FEB 2019 | NORTH AMERICAN | SPECTRUM.IEEE.ORG get a decent recording from the lungs, the health worker had to position the device’s chest piece in just the right places on the child’s chest and back. Unfortunately, the local workers typi- cally had only basic training in how to use the tool, and often missed the “sweet spots” that best transfer the lung sounds to the stethoscope head. Furthermore, they were usually making their record- ings in noisy settings, nothing like the quiet doctor’s ofce for which the stetho- scope is intended. Background sounds of people, machines, cars, and motor- cycles would mix with the lung sounds, leaving the expert with a recording of cacophony. In addition, the local work- ers needed Internet access to transmit their fles, and connectivity limitations and glitches meant that it often took many hours or even days to get back an expert diagnosis. If they wanted to get useful record- ings, the Johns Hopkins doctors decided, they’d have to entirely rethink the stethoscope. So they approached the university’s department of electrical and computer engineering, where one of us (West) is a professor, and the other (Elhi- lali) is an associate professor. As both of us specialize in audio engineering, we were eager to take up the challenge.

ERE’S HOW A CLASSIC s t e t h o s c o p e w o r k s . Sound is produced by an WORKING PROTOTYPES: After an early version of the smart stethoscope [top] proved organ in the body, and effective at filtering out noise and giving the listener a clear signal from the lungs, Johns Hopkins researchers developed the current version [bottom], which includes an LED dis- these acoustic waves play that gives the user a recommendation on the diagnosis. Hcause a vibration in the stethoscope’s chest piece, which acts as a resonator. This chest piece has two sides with carefully chosen shapes—a flat, disk- the user’s ears, sound levels are usu- phragm” and “bell” modes, and can like diaphragm and a hollow cup called ally quite low. often adjust the sounds that reach their a bell—that naturally oscillate with dif- More-modern electronic stetho- ears in other ways. ferent frequency ranges, and which are scopes convert the acoustic wave into The Johns Hopkins device is an elec- used for diferent diagnostic tasks. The electric signals that can be processed in tronic stethoscope that improves on digi- acoustic vibration travels via an air- the device to amplify the sounds. They tal devices currently on the market. The filled tube that connects to two ear- replace the double-sided chest piece upgrades start with its hardware: The pieces, thus relaying the sounds of the with one gadget that uses transducers chest piece is packed with transducer patient’s body to the listener. Although to convert acoustic signals to electric arrays to achieve a uniform sensitivity the stethoscope is designed to max- ones with minimal distortion or noise over the entire active area. This design

TOP: JOHNS HOPKINS UNIVERSITY; BOTTOM: ZEBADIAH POTLER/JOHNS HOPKINS UNIVERSITY HOPKINS POTLER/JOHNS ZEBADIAH BOTTOM: UNIVERSITY; HOPKINS JOHNS TOP: imize sound pickup and delivery to efects. Users can toggle between “dia- delivers a strong signal even when the

SPECTRUM.IEEE.ORG | NORTH AMERICAN | FEB 2019 | 39 chest piece isn’t placed in precisely SEEKING CLARITY: Traditional stethoscopes are designed for quiet doctors’ offices, the right location, which gives a major but many health clinics around the world are far from quiet. The devices are also meant to be used by experts trained in the subtle art of interpreting body sounds. The assist to untrained users. We’ve experi- smart stethoscope provides a clear audio signal by canceling out both noise from the mented with a variety of transducers, environment and irrelevant noise from inside the body. including microelectromechanical sys- Ambient noise tems (MEMS), which allow us to pack a great many microphones into a small 2 area, as well as nanofber materials that match the acoustic properties of skin 1

to limit signal loss. Kilohertz 0 For a stethoscope to be useful in clin- 0 12 34 5 ics in the developing world, dealing with Seconds noise is of utmost importance. In clas- [1] The smart scope has an external microphone that records ambient sounds. The sic stethoscopes, environmental noise system uses active and adaptive acoustics to remove those sounds, in real time, from contaminates the signal at three points: the digitized audio signal transmitted to the user’s ears. at the chest piece, through the rubber hose, and at the ear of the user. The Noisy body sounds Johns Hopkins design mitigates noise 2 by improving the coupling between

the patient’s body and the chest piece, 1 swapping the rubber hose for an elec- Kilohertz tric cable, and employing digital noise- 0 control techniques to ensure that a 0 12 34 5 strong signal arrives at the user’s ears. Seconds Our stethoscope also has an external [2] Pulmonologists are trained to ignore other body sounds, such as the heartbeat, microphone in the chest piece to collect when they listen to the lungs. The smart scope detects those extraneous sounds and removes them, too. ambient sounds. This data can then be used to reduce noise through adaptive Enhanced signal signal analysis—as we’ll explain in the next section. 2 Our stethoscope gets its smarts from an onboard microprocessor that serves 1

as an operating system, allowing us to Kilohertz develop various applications to tailor 0 the device for different needs. So far, 0 12 34 5 Seconds our team has focused on two apps that we deemed crucial for developing world [3] What’s left is a clear audio signal containing only the sounds from the patient’s lungs. With this refined signal, both the listener and the stethoscope’s software can identify clinics. One dials down the level of noise; the signs of pneumonia. the other dials up the level of expertise.

EDUCING EXTERNAL a child is wheezing, producing high- As we designed our noise-cancellation noise is a complicated pitched whistling sounds in addition algorithm, we considered two challenges. matter. It’s relatively easy to the normal sounds of breathing, the First, the lung sounds that indicate pos- to design noise-canceling “de-noising” algorithm should not mis- sible pathologies are unpredictable and software that recognizes take this whistling for high-pitched irregular. Second, the noises in a busy Rthe signal pattern created by nor- ambient noise, such as a crying child clinic are highly variable: The exam room mal breathing and filters out every- would make. Conversely, trying to pre- could be flled with chattering people, thing else. But pathological conditions serve abnormal patterns can also pre- ringing phones, and whirring fans; the like pneumonia create abnormal serve ambient noise. Our app aims to child being examined might be crying signal patterns—and an overactive suppress undesired noise while main- out and thrashing around on the exam noise-canceling flter can remove that taining the integrity of the signal com- table; and the movements of the stetho-

critical information. For example, if ing from the patient’s lungs. scope itself can produce electronic noise. UNIVERSITY HOPKINS JOHNS

40 | FEB 2019 | NORTH AMERICAN | SPECTRUM.IEEE.ORG With those challenges in mind, our Our AI approach was inspired by the gists and gastroenterologists to collect team created a noise-cancellation algo- brain’s auditory system, which ana- training data and design algorithms. rithm that uses active and adaptive lyzes both the overall audio signal and Today’s electronic stethoscopes typ- acoustics. Instead of just setting up a fl- its specifc features. The brain devotes ically cost around US $500, making ter and letting it run in a passive way, our the most processing power to the fre- them far too expensive for many health system analyzes both lung sounds and quencies that change over time and workers in the developing world. The ambient sounds recorded by the exter- those that it judges most relevant to Johns Hopkins smart scope is designed nal microphone, looking at their fre- the situation. Think about being at a to be signifcantly cheaper, with aford- quencies on the audio spectrum. Then noisy cocktail party, and how you’re able electronic components and low- the app adapts the degree and spectral able to ignore the background buzz of cost power and computing options, span of noise cancellation as the algo- chatter to focus on the words spoken in hopes that it can be useful for low- rithm tracks the two signals over time. by the person right in front of you. As resource communities. We validated this approach through a our app analyzes the de-noised lung Right now, our smart scope is under- blind listening test, in which we asked sound, it takes in the whole signal, but going extensive feld testing. Clinic work- a panel of pediatric pulmonologists to zeroes in on the specifc modulations ers in Peru, Bangladesh, and Malawi are compare the quality of sounds coming that indicate a case of pneumonia. trying out prototypes, as are doctors at from a commercial stethoscope used in We used a machine-learning approach the Johns Hopkins pediatric emergency a rural clinic with the same signals run to train the app, giving it plenty of data room. These testing and validation stud- through the Johns Hopkins algorithm. and a mandate to pick out the most ies are assessing how well it functions in The experts preferred our de-noised useful features. The algorithm trained noisy settings and how accurate its pneu- signal 95 percent of the time. on a database of recordings of about monia diagnoses are across a diverse We’ve also extended the audio soft- 1,500 patients from fve African coun- population of patients. In these stud- ware to deal with extraneous sounds tries and two Asian countries. When ies, we’re evaluating the lung sounds from the body. When a health worker we tested the AI-enabled app, we found using objective signal quality metrics, is trying to listen to a patient’s lungs, that it could automatically distinguish and we’re validating the diagnoses by the sound of a beating heart can be dis- healthy people from those with pneu- comparing them with diagnoses made tracting and can mask the lung sounds. monia with an accuracy of 87 percent, via chest X-rays and the listening assess- Pulmonologists train their ears to far surpassing other automatic diag- ments of pulmonology experts. ignore the heartbeat and focus on the nosis methods. Our team is continuing While we continue our work in aca- breathing patterns—and we gave our to work on this algorithm, aiming to demia, some of our colleagues have algorithm similar training. Our app improve its performance with machine- founded a startup called Sonavi Labs to cancels out the sound of the heartbeat learning techniques. quickly turn our prototype into a com- to provide the user with the pure and Health workers in rural clinics could mercially viable product. The mission unadulterated signal from the lungs. use our smart stethoscope even if they of Sonavi Labs is to develop modern don’t have access to the Internet; the medical devices that effectively use onboard technology does all the pro- lung sounds for the diagnosis of dis- O SAVE THE LIVES OF cessing and provides an instant recom- ease. Their frst products, digital stetho- children throughout the mendation on diagnosis using a small scopes called the Feelix and FeelixPro, developing world, we wish built-in LED display. We envision that are set to launch in early 2019. we could send expert physi- the final version of the smart scope By enabling on-the-spot screenings cians to staf every remote will connect via Bluetooth to the user’s with a cheap device that can be handled Tclinic and overcrowded hospital. So we phone or tablet, where the user can get by local health workers, we hope our did the next-best thing and developed more information about the diagnosis, reengineered scope will make a difer- an app to send physician expertise to visualize the lung-sound signal, and ence in the global health crisis of child- these settings. play back previously recorded sounds. hood pneumonia. Viewed on the macro This second app, the one that makes As our smart stethoscope has a pro- level, we hope to scale up our eforts so the Johns Hopkins stethoscope truly grammable platform, we’re now work- that hundreds of thousands of lives can smart, automatically screens for cases ing to develop additional apps that be saved. But on the micro level, we feel of pneumonia by distinguishing nor- would make it a multipurpose tool, use- that if even one parent sees her child mal from abnormal breathing patterns, ful for many diseases and scenarios. We recover because of early diagnosis, all looking in particular for the wheezing think it could be useful in diagnosing our hard work will have been worth it. ■ or crackling sounds that can indicate heart failure and various gut disorders, ↗ POST YOUR COMMENTS at https://spectrum.ieee.org/ liquid and infammation in the lungs. and we’re collaborating with cardiolo- stethoscope0219

SPECTRUM.IEEE.ORG | NORTH AMERICAN | FEB 2019 | 41