Article An Intelligent Baby Monitor with Automatic Sleeping Posture Detection and Notification Tareq Khan School of Engineering, Eastern Michigan University, Ypsilanti, MI 48197, USA; [email protected] Abstract: Artificial intelligence (AI) has brought lots of excitement to our day-to-day lives. Some examples are spam email detection, language translation, etc. Baby monitoring devices are being used to send video data of the baby to the caregiver’s smartphone. However, the automatic understanding of the data was not implemented in most of these devices. In this research, AI and image processing techniques were developed to automatically recognize unwanted situations that the baby was in. The monitoring device automatically detected: (a) whether the baby’s face was covered due to sleeping on the stomach; (b) whether the baby threw off the blanket from the body; (c) whether the baby was moving frequently; (d) whether the baby’s eyes were opened due to awakening. The device sent notifications and generated alerts to the caregiver’s smartphone whenever one or more of these situations occurred. Thus, the caregivers were not required to monitor the baby at regular intervals. They were notified when their attention was required. The device was developed using NVIDIA’s Jetson Nano microcontroller. A night vision camera and Wi-Fi connectivity were interfaced. Deep learning models for pose detection, face and landmark detection were implemented in the microcontroller. A prototype of the monitoring device and the smartphone app were developed and tested successfully for different scenarios. Compared with general baby monitors, the proposed device gives more peace of mind to the caregivers by automatically detecting un-wanted situations. Keywords: human pose detection; face detection; deep learning; cloud messaging; hypertext transfer Citation: Khan, T. An Intelligent protocol (HTTP) server; jetson nano Baby Monitor with Automatic Sleeping Posture Detection and Notification. AI 2021, 2, 290–306. https://doi.org/10.3390/ai2020018 1. Introduction Smart baby monitoring devices are being used to obtain and send video and audio Academic Editor: Giovanni Diraco data of the baby to the caregiver’s smartphone, but most of these devices are unable to recognize or understand the data. In this project, a novel baby monitoring device is Received: 27 April 2021 developed which automatically recognizes the undesired and harmful postures of the baby Accepted: 15 June 2021 Published: 18 June 2021 by image processing and sends an alert to the caregiver’s smartphone—even if the phone is in sleep mode. Deep learning-based object detection algorithms are implemented in the Publisher’s Note: MDPI stays neutral hardware, and a smartphone app is developed. The overall system is shown in Figure1. with regard to jurisdictional claims in The research problem addressed in this paper is to develop methods to automatically published maps and institutional affil- detect (a) whether the baby’s face is covered due to sleeping on the stomach; (b) whether the iations. baby has thrown off the blanket from the body; (c) whether the baby is moving frequently; (d) whether the baby’s eyes are opened due to awakening. One of the challenges of deep learning models is running them in embedded systems where resources such as memory and speed are limited. The methods must work in an embedded system with low latency. The system should also work in both day and night conditions. The detection methods Copyright: © 2021 by the author. Licensee MDPI, Basel, Switzerland. should not be biased and be inclusive to all races of babies. This article is an open access article The objective of this study is to develop a baby monitoring device that will auto- distributed under the terms and matically recognize the harmful postures of the baby by image processing and send an conditions of the Creative Commons alert to the caregiver’s smartphone. The work will be considered successful when the Attribution (CC BY) license (https:// proposed baby monitor can automatically detect the targeted harmful postures and send a creativecommons.org/licenses/by/ notification to the smartphone. Experiments with different postures will be conducted and 4.0/). the latency of the detection algorithms will be measured. AI 2021, 2, 290–306. https://doi.org/10.3390/ai2020018 https://www.mdpi.com/journal/ai AI 2021, 2, FOR PEER REVIEW 2 baby monitor can automatically detect the targeted harmful postures and send a notifica- AI 2021, 2 tion to the smartphone. Experiments with different postures will be conducted291 and the latency of the detection algorithms will be measured. FigureFigure 1. 1.The The overall overall smart smart baby baby monitoring monitoring system: system: (a) baby (a) sleeping;baby sleeping; (b) smart (b) baby smart monitoring baby monitoring devicedevice automatically automatically detects detects harmful harmful postures postures such as such facecovered, as face covered, thrown off thrown the blanket, off the frequently blanket, fre- movingquently or moving awake; (cor) imageawake; data (c) are image sent todata the are smartphone sent to the through smartp thehone Internet through with the the message Internet of with the anymessage harmful of situation; any harmful (d) a real-timesituation; video (d) a of real-time the baby is video shown of in the the baby smartphone, is shown and in notifications the smartphone, andand alerts notifications are generated and alerts whenever are generated the caregiver whenever receives the a message caregiver of receives the harmful a message situation of of the harm- theful baby.situation of the baby. TheThe needs needs and and significances significance ofs the of proposedthe proposed system system are mentioned are mentioned below: below: • About 1300 babies died due to sudden infant death syndrome (SIDS), about 1300 deaths • About 1300 babies died due to sudden infant death syndrome (SIDS), about 1300 were due to unknown causes, and about 800 deaths were caused by accidental suffoca- tiondeaths and strangulationwere due to unknown in bed in 2018 causes, in the and USA ab [1out]. Babies 800 deaths are at higherwere caused risk for SIDSby accidental ifsuffocation they sleep on and their strangulation stomachs as it in causes bed in them 2018 to in breathe the USA less air.[1]. The Babies best are and at only higher risk positionfor SIDS for if a they baby tosleep sleep on is their on the stomachs back—which as it the causes American them Academy to breathe of Pediatrics less air. The best recommendsand only position through for the a baby’s baby firstto sleep year [is2]. on Sleeping the back—which on the back improvesthe American airflow. Academy Toof reduce Pediatrics the risk recommends of SIDS, the baby’s through face the should baby’s be uncovered, first year and[2]. bodySleeping temperature on the back im- shouldproves be airflow. appropriate To reduce [3]. The the proposed risk of baby SIDS, monitor the baby’s will automatically face should detect be uncovered, these and harmfulbody temperature postures of the should baby andbe appropriate notify the caregiver. [3]. The This proposed will help baby to reduce monitor SIDS. will auto- • Babies—especiallymatically detect fourthese months harmful or older—move postures of frequentlythe baby and during notify sleep the and caregiver. can throw This will off the blanket from their body [4]. The proposed system will alert when the baby is help to reduce SIDS. • moving frequently and also whether the blanket is removed. Thus, it helps to keep theBabies—especially baby warm. four months or older—move frequently during sleep and can • Babiesthrow may off wakethe blanket up in the from middle their of body the night [4]. The due proposed to hunger, pain,system or justwill toalert play when the withbaby the is parent. moving There frequently is an increasing and also call whether in the medical the blanket community is removed. to pay attention Thus, it helps to tokeep parents the when baby they warm. say their babies do not sleep [5]. The smart baby monitor detects • whetherBabies themay baby’s wake eyes up arein openthe middle and sends of the an alert. night Thus, due itto helps hunger, the parents pain, or know just to play whenwith the the baby parent. is awake There even is an if increasing he/she is not call crying. in the medical community to pay attention • Whento parents a baby when sleeps they ina say different their babies room, do the no caregiverst sleep [5]. need The to smart check baby the sleeping monitor detects condition of the baby after a regular interval. Parents lose an average of six months’ whether the baby’s eyes are open and sends an alert. Thus, it helps the parents know sleep during the first 24 months of their child’s life. Approximately 10% of parents managewhen the to getbaby only is 2.5awake h of continuouseven if he/she sleep is eachnot crying. night. Over 60% of parents with • babiesWhen aged a baby less thansleeps 24 monthsin a different get no more room, than the 3.25 caregivers h of sleep eachneed night. to check A lack the of sleeping sleepcondition can affect of the qualitybaby after of work a regular and driving; interval. create Parents mental healthlose an problems, average such of six as months’ anxietysleep during disorders the and first depression; 24 months and causeof their physical child’s health life. problems,Approximately such as obesity,10% of parents highmanage blood to pressure, get only diabetes, 2.5 h of and continuous heart disease sleep [6]. each The proposed night. Over smart 60% device of parents will with automaticallybabies aged detectless than the 24 situations months when get no the more caregiver’s than 3.25 attention h of sleep is required each night. and A lack of sleep can affect the quality of work and driving; create mental health problems, AI 2021, 2, FOR PEER REVIEW 3 such as anxiety disorders and depression; and cause physical health problems, such as obesity, high blood pressure, diabetes, and heart disease [6].