Zhou et al. Cardiovasc Ultrasound (2021) 19:29 https://doi.org/10.1186/s12947-021-00261-2

REVIEW Open Access Artifcial intelligence in : detection, functional evaluation, and disease diagnosis Jia Zhou1, Meng Du2, Shuai Chang1 and Zhiyi Chen1,2*

Abstract Ultrasound is one of the most important examinations for clinical diagnosis of cardiovascular diseases. The speed of image movements driven by the frequency of the beating is faster than that of other organs. This particularity of echocardiography poses a challenge for sonographers to diagnose accurately. However, artifcial intelligence for detection, functional evaluation, and disease diagnosis has gradually become an alternative for accurate diagnosis and treatment using echocardiography. This work discusses the current application of artifcial intelligence in echocar- diography technology, its limitations, and future development directions.

Highlights 1. Application of artifcial intelligence (AI) in echocardiography is now widely studied, and AI technique has the poten- tial to optimize the diagnostic potential of echocardiography. 2. Application of artifcial intelligence in echocardiography is important in the following aspects: recognizing the standard section, cardiac cavity automatic segmentation, functional left assessment, and cardiac disease diagnosis. 3. Standardized data collection and image annotation are essential for artifcial intelligence in echocardiography. Keywords: Artifcial intelligence (AI), Echocardiography, Segmentation, Functional evaluation, Normalization

Background [4]. In the diagnosis and treatment of heart diseases, AI Te application of artifcial intelligence (AI) technol- techniques have been applied to , ogy in cardiovascular imaging has become a research vectorcardiography, echocardiography, and electronic hotspot in recent years, as it may reduce treatment cost health records [5]. As a non-invasive imaging detection and help avoid unnecessary testing [1]. AI technology method for cardiac structure and functional evaluation, has been progressively applied for processing multiple echocardiography technology has certain limitations. modal images, such as auxiliary electrocardiograph diag- Tese include a long procedure time (more than 20 min, nosis [2], cardiac computerized tomography (CT) detec- even if no abnormalities are detected), multiple measure- tion [3], and radionuclide myocardial perfusion imaging ment values that increase the duration complexity and user subjectivity, complex analyses during the evalua- *Correspondence: [email protected] tion, high standard of individualized assessments [6], 1 The First Afliated Hospital, Medical Imaging Centre, Hengyang high operator subjectivity, and wide observation ranges Medical School, University of South China, 69 Chuanshan Road, Hengyang 421001, China and distinctions among observers that persist even under Full list of author information is available at the end of the article standardized conditions. Tese limitations also lead to a

© The Author(s) 2021. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creat​ iveco​ mmons.​ org/​ licen​ ses/​ by/4.​ 0/​ . The Creative Commons Public Domain Dedication waiver (http://creat​ iveco​ ​ mmons.org/​ publi​ cdoma​ in/​ zero/1.​ 0/​ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Zhou et al. Cardiovasc Ultrasound (2021) 19:29 Page 2 of 11

high demand for medical specialist training in the feld of functional evaluation, and auxiliary disease diagnosis echocardiography. In recent years, the application of AI faster and more accurate [10, 11] (Fig. 1). Although a for diagnosis and treatment using echocardiography has series of articles published in high-level journals posi- been shown to have the potential to solve these problems. tively afrm the role of AI technology in diagnosis of Te integration of echocardiography and AI is not echocardiography [16–18], there are remaining concerns, a brand new topic. Earlier cases of integrated applica- such as insufcient standardization of echocardiography, tion of echocardiography and machine learning can poor robustness, and insufcient generalization of the be traced back to 1978 when Fourier analysis was used models in clinical applications. Tis review summarizes to evaluate the waveform of anterior mitral leafets via the application (Table 1) and advantages of echocardiog- M-mode ultrasound. Studies have confrmed that this raphy integrated with AI, analyzes the associated limita- method had a remarkable impact on auxiliary diagnosis tions, and systematically investigates the future trends of of mitral valve prolapse [7]. Machine learning is a sig- AI technology in echocardiography from the perspective nifcant AI method. Before deep learning was proposed of practical applications. in 2006, plenty of machine learning algorithms had been applied to echocardiographic evaluation of cardiac Main text function, image optimization, and structural observa- Standard section recognition with assistance of AI tion in the form of software or cutting edge technology, technology such as semi-automatic speckle tracking technology Te anatomical structure of the heart is complex, and and the Simpson method. In early 2020, the Food and sonogram genres are disparate. Terefore, it is particu- Drug Administration announced that it had authorized larly necessary to manage the recognition and functional Caption Guidance software from Caption Health to be evaluation of the cardiac structure in diferent models. available for sale to collect data from echocardiographic However, there are numerous standardized sections of images [8] (https://​capti​onhea​lth.​com/). Te develop- echocardiography. Patients without abnormalities have ment of novel technologies, such as deep learning and to undergo 10–20-section scans. Occasional slight angle neural networks, has efectively improved the efcacy of diferences among sections cause extreme difculties echocardiography [9], making standard section identif- in identifcation of various sections for physicians with cation of cardiac anatomical structures, automatic rec- insufcient experience and qualifcations, resulting in ognition and segmentation of cardiac structures, cardiac failure to provide accurate and standardized analysis. It

Fig. 1 Application of artifcial intelligence in echocardiography [12–15] Zhou et al. Cardiovasc Ultrasound (2021) 19:29 Page 3 of 11 J Am Soc Echocardiogr, 2017 J Am Soc Echocardiogr, [ 25 , 26 ] Trans Ultrason Ferroelectr Freq Freq Ultrason Ferroelectr Trans 2015 [ 23 , 24 ] Control, Biomed, 2014 [ 20 ] Biomed, Circ Cardiovasc Imaging, 2021 [ 31 ] Imaging, Cardiovasc Circ JACC Cardiovasc Imaging, 2021 [ 30 ] Imaging, Cardiovasc JACC J Am Soc Echocardiogr, 2020 [ 29 ] J Am Soc Echocardiogr, Circ Cardiovasc Imaging, 2019 [ 28 ] Imaging, Cardiovasc Circ J Am Coll Cardiol, 2015 [ 27 ] Cardiol, J Am Coll JACC: Cardiovascular Imaging, 2016; Imaging, Cardiovascular JACC: Biomed Mater Eng, 2014 [ 13 ] Biomed Mater Eng, Phys Med Biol, 2013 [ 22 ] Med Biol, Phys Ultrasound Med Biol, 2015; IEEE Ultrasound Med Biol, Ultrasound Med Biol, 2014 [ 21 ] Ultrasound Med Biol, Comput Methods Programs MethodsComput Programs Biomed Mater Eng, 2014 [ 13 ] Biomed Mater Eng, NPJ Digital Med, 2018 [ 12 ] Med, Digital NPJ Ultrasound Med Biol, 2019 [ 19 ] Ultrasound Med Biol, Reference 0.86 – 0.95 a = = 0.98 r = 83.3% ~ 91.3% (LV) 90.8% (epicardial bound - 90.8% (epicardial = = 0.92 = 97% ~ 98% = 97.8% = 98.3%

= 1.0 (AI and physician), 0.84 (novice 0.84 (novice and physician), (AI using AI and physician) ary), 87.3% (endocardial boundary) = 0.95 = 0.87 ~ 0.96/ Intraclass correlation ACC AUC r ICC r N/A Dice Mean dice N/A N/A N/A ACC ACC Diagnostic EfciencyDiagnostic thickness constraint feature ing technology algorithm Snake Neural network PWC-Net 3D CNN AutoEF (BayLabs) AutoEF AutoLV (TomTec Imaging system) (TomTec AutoLV HeartModel (Philips) Ant colony optimization Ant colony Sparse matrix transform/ wall Sparse matrix transform/ Active shape model/ fusion-imag - Active Iterative 3-D cross-correlation Radial method active contour Ant colony optimization Ant colony CNN CNN Detail method/software AI algorithm AI algorithm AI algorithm Commercial technique Commercial Commercial technique Commercial Commercial technique Commercial AI algorithm AI algorithm AI algorithm AI algorithm AI algorithm AI algorithm AI algorithm AI algorithm AI algorithm or commercial technique segmentation (provide accuracy improvement accuracy(provide improvement boundaryof the endocardial recognition) views Assessment of leftAssessment heart function Right segmentation ventricular Atrial and multi-chamber heart Left ventricular segmentation segmentation ventricular Left Automatic classifcation of cardiac classifcation of cardiac Automatic Aim of study Aim Previous AI studies of echocardiography Previous image acquisition 1 Table LV assessment LV Segmentation Section recognition/ Section recognition/ Zhou et al. Cardiovasc Ultrasound (2021) 19:29 Page 4 of 11 [ 45 ] Soc Echocardiogr, 2012 [ 32 , 33 ] Soc Echocardiogr, J Ultrasound Med, 2014 [ 46 ] J Ultrasound Med, IEEE, 2016 [ 44 ] Comput MedComput Imaging Graph, 2006 Ultrasound Med Biol, 2017 [ 15 ] Ultrasound Med Biol, Echocardiography, 2015 [ 43 ] Echocardiography, AMIA Annu Symp Proc, 2014 [ 41 ] Annu Symp Proc, AMIA JACC Cardiovasc Imaging, 2019 [ 42 ] Imaging, Cardiovasc JACC Circ Cardiovasc Imaging, 2016 [ 40 ] Imaging, Cardiovasc Circ Nat Commun, 2021 [ 38 ] Nat Commun, Circ J, 2019 [ 39 ] J, Circ Int J Cardiovasc Imaging, 2019 [ 37 ] Imaging, Int J Cardiovasc J Am Soc Echocardiogr, 2016 [ 36 ] J Am Soc Echocardiogr, Echocardiography, 2016 [ 35 ] Echocardiography, Int J Cardiovasc Imaging, 2015 [ 34 ] Imaging, Int J Cardiovasc Circ Cardiovasc Imaging, 2013; J Am Imaging, Cardiovasc Circ Reference a = 67%, = 79% = 0.955 = 0.970 = 94% = 77% = 91% = 0.96 0.85 and 0.65 = 74% ~ 77% = 77.8% = 90% ~ 99% = 89.2% = = 89%

PPV ity cients = 0.67 ~ 0.99 = 0.93 Specifcity Mean sensitivity Accuracy Highest accuracy Sensitivity Specifcity r AUC AUC AUC Sensitivity (Recall) N/A Intra- and interobserver variabil- ACC N/A r Intraclass coef - correlation Diagnostic EfciencyDiagnostic Ventricular Left - ANN (Classifca based features/ tion) quantifcation texture feature analysis feature texture sound (AIUS) Doppler (RT-VCFD) (PISA) Gray level co-occurrence level Gray matrix- Feed-forward neural network Feed-forward Texture analysis Texture A specialised software for MCE A specialised software for EchoPAC PC (GE) EchoPAC Support Vector Machine discrete wavelet transform and transform wavelet discrete Associative memoryAssociative classifer 2D/3D CNN Myocardial strain analysis Myocardial Anatomical afne optical fow Anatomical Anatomical IntelligenceAnatomical in ultra - Mitral Valve Navigator (Philips) Navigator Valve Mitral Real-time 3D volume color-fow color-fow Real-time 3D volume Proximal isovelocity surface isovelocity area Proximal Detail method/software AI algorithm AI algorithm AI algorithm AI algorithm Commercial technique Commercial AI algorithm AI algorithm AI algorithm AI algorithm Commercial technique Commercial AI algorithm Commercial technique Commercial Commercial technique Commercial Commercial technique Commercial Commercial technique Commercial AI algorithm or commercial technique Area under the curve, ANN Artifcial Area Network, Neural LV ​ AUC ​ Accuracy, disease Intracardiac masses Coronary atherosclerotic heartCoronary atherosclerotic Cardiomyopathy Valvular heart disease Valvular Aim of study Aim (continued) Diagnostic Efciency: including recognition/diagnostic accuracies, correlation coefcient, PPV (positive predictive value), sensitivity (recall) and specifcity, etc. and specifcity, sensitivity value), (recall) predictive PPV (positive coefcient, correlation Efciency: Diagnostic accuracies, including recognition/diagnostic 1 Table network, neural ACC CNN Convolutional a Cardiac disease diagnosis Cardiac Zhou et al. Cardiovasc Ultrasound (2021) 19:29 Page 5 of 11

typically takes at least 1 to 2 years for a sonographer to variability can also be substandard when the functional grasp the basic echocardiography principles and meth- left ventricle assessment is performed by a person. ods of operation. In China, many grassroots regions can- Previous research on left ventricular function assess- not aford the labor cost of training echocardiography ment with assistance of AI technology has been mainly physicians. Terefore, two-dimensional (2D) ultrasound based on automatic segmentation of the left ventricle data sets from more than 500 patients and 7,000 videos and endocardial tracking technology [13]. At present, were gathered from research performed in 2018. A clas- there are several commercial software packages that sifcation model was then constructed using the method can achieve high-accuracy 2D and 3D echocardiogra- of convolutional neural networks by dividing them into phy measurements, which further realize the automated seven groups of various cardiac views. Te model’s clas- assessment of the left heart function. One of the com- sifcation accuracy reached 98% [19]. Likewise, research- monly used software is the Philips EPIQ series for tran- ers from the University of California, San Francisco used sthoracic 3D echocardiography left ventricular cavity echocardiographic images from 267 patients to classify quantitative system HeartModel (Philips, Eindhoven, static and dynamic original images using deep learning Netherlands), which utilizes an adaptive analysis algo- approaches and to build an automatic section recogni- rithm [25, 26]. AutoLV (TomTec Imaging System, Ger- tion model with classifcation criteria from 15 standard many) is an additional standard tracking system for left sections. Te outcome showed that recognition accu- ventricular ejection fraction and longitudinal strain. Vari- racy achieved 97.8%, while physician recognition accu- ous clinical studies have shown that automatic software racy was approximately 70.2–83.5% [12]. Rapid standard used to assess the ventricular volume and ejection frac- section recognition with AI technology can somewhat tion can provide accuracy that is similar to manual meth- shorten the evaluation time and enhance detection ability ods, which has a good correlation with cardiac MRI [27]. as well as novice accuracy. Tis technology has the poten- In spite of this, boundary identifcation is still prone to tial to be applied in specialized non-echocardiography or errors limiting accuracy. With the development of auto- emergency departments, which would advance opera- mated technique, some researchers have tried to reduce tional efciency. It can also be extended to the grassroots these errors by using evaluation methods without vol- regions where echocardiography physician resources for ume measurements. Tese algorithms mimicked what standardized inspection and quality control are scarce. an experienced and brain can do, instead of tracing the endocardial borders and calculating ventricu- Functional left ventricle assessment with assistance lar volumes [28, 29]. Since the analyses of GLS are time from AI technology consuming and demand expertise, AI technique can help Functional evaluation of the left ventricle is one of the identify the standard apical views, perform timing of most important and routine examination procedures in cardiac events, trace the myocardium, perform motion echocardiographic diagnosis. Functional evaluation indi- estimation, and measure GLS in less than 15 s. It was cators of the left ventricle systole include left ventricular prove to have highly signifcant correlation with a con- ejection fraction (EF), left ventricular volume, left ven- ventional speckle-tracking application [30]. Furthermore, tricular wall motion function, myocardial contractility AI technology may help novices quickly acquire skills and global longitudinal strain (GLS). EF is the most con- in quality diagnostic imaging to improve the inter- and venient and commonly used indicator for evaluating left intra-observer variability [31, 47]. For the prognosis, a ventricular systolic function. Since EF is a ratio when no novel multicenter research demonstrated that AI-based segmental motion abnormality is present in the ventricu- LV analyses were signifcant predictors of mortality, lar wall, EF can be determined using an M-mode chart by which is better than manual measurement. It could mini- measuring the inside diameter ratio of the left ventricular mize variability of quantifcation of LVEF and LVLS [20]. end diastole to systole. A more precise approach for EF AI can be a game-changer in this feld, providing a repro- measurement is the biplane Simpson method, especially ducible LVEF evaluation that is independent of human when segmental motion abnormalities such as myocar- observer. dial occur in the ventricular wall. Te M-mode of certain left ventricular sections cannot represent the Automatic segmentation of cardiac cavity with assistance motion of the entire left ventricle. At this time, it is nec- of AI technology essary to estimate the overall volume using the Simpson Te shape and function of the four chambers of the heart method. Regardless of the method employed, evalua- (left/right ventricle, left/right ) are observed fol- tions rely on visual observation and manual boundary lowing the determination of the cardiogram section. tracing, while repeatability and accuracy depend on the If the heart morphology is afected by certain disease- physician’s experience [27]. Te inter- and intra-observer related factors, the standard pressure and volume will Zhou et al. Cardiovasc Ultrasound (2021) 19:29 Page 6 of 11

change, resulting in cardiac chamber enlargement, com- recognition will prompt the precision of the automatic pensatory wall thickening, and cardiac remodeling [21]. left ventricular segmentation. Terefore, it is important to obtain accurate segmenta- tion of cardiac ultrasound images and understand the Right ventricular segmentation morphological changes for clinical diagnosis. Masses of Compared to left ventricular segmentation, right ven- echocardiography instruments are equipped with semi- tricular segmentation presents a relatively difcult- automatic cardiac chamber segmentation software. to-resolve problem. Tese problems are related to the Manual segmentation is tedious, time-consuming, and characteristics of the right ventricle and its wall, includ- subjective. Hence, automatic and precise segmentation ing complex crescent-shaped structure, presence of can decrease the occurrence of the above-mentioned trabecular myocardium, thinner and weaker right ven- problems and has favorable clinical values. At present, tricular walls, irregular endocardium shapes and edges cardiac chambers are automatically segmented by recog- on cardiogram images, and relatively poor image qual- nizing the endocardial wall in 2D or three-dimensional ity due to its behind-sternum, location leading to lung (3D) images, which is common in segmentation of the gas irruption and shadow of the sternum. Tese cause left and right ventricles. During segmentation, the auto- blurred ventricular wall echo and even disappearance of matic evaluation and accurate measurement of parame- entire lateral walls in some images. However, accurate ters such as cardiac cavity size can also be accomplished. evaluation of right ventricular function is signifcant for functional analysis of the , surgery Left ventricular segmentation selection of congenital heart disease (CHD), and predic- Left ventricular segmentation is popular in the automatic tion and evaluation of heart failure. Te precise identif- segmentation of the heart chamber. Te accurate meas- cation and segmentation of right ventricular ultrasound urement of ejection fraction and evaluation of the move- images can quickly and efciently capture the diameter, ment in left ventricular myocardium can be achieved by area, and volume of the right ventricle, myocardial thick- segmenting the left ventricle. Two- and three-dimen- ness, fractional area changes, as well as other indicators sional ultrasound methods are both widely used in the to provide more information for auxiliary clinical diag- assessment of left ventricular segmentation. Compared nostics. To overcome these issues, Qin et al. proposed to 2D ultrasound images, 3D-image resolution is lower. an automatic segmentation framework based on the Manual processing and analysis are also extremely time- sparse matrix transform and introduced a wall thickness consuming. Fully automatic left ventricular segmentation constraint feature. A positive segmentation result was based on ultrasound images remains a challenging task acquired via detection of endocardium and epicardium at due to rapid and large-scale myocardial movements, res- the same time when the lateral walls are blurred [23]. In piratory interference, inconsistent motion between mitral addition, Bersvendsen et al. proposed and constructed a valve opening and closing, in addition to inherent noise multi-chamber model based on how the chambers inter- and artifacts in ultrasound imaging. Some methods have act while performing the pumping function, which allows been proposed to address the above-mentioned issues. for coupled segmentation of endo- and epicardial bor- One solution evaluated the ventricular cavity func- ders of the left and right ventricle. Te establishment of tion based on information such as variance to prioritize the multi-chamber model can allow for a complete clini- endocardial boundary refnement [13], followed by con- cal condition evaluation [24]. In general, when the image tinuously tracking and measuring endocardial boundary quality is poor, the precision of right ventricular segmen- deformation during the cardiac cycle. Te other approach tation is dependent on previously obtained information is to apply the radial active contour method Snake to seg- and correlation with other heart structures, such as myo- ment the ventricle from the short-axis view instead of cardium, epicardium, and left ventricle. the long-axis view with a large deformation. Tis method is less infuenced by noise and artifacts and has a more Atrial and multi‑chamber heart segmentation solid robustness [22]. In addition, for the automatic seg- Echocardiographic atrial segmentation has certain appli- mentation of the left ventricle in a 3D echocardiogram, cation values in minimally invasive interventional treat- segmentation performance can be efectively boosted by ment of arrhythmia such as cardiac . adding the maximum cross-correlation (MCC) values of For example, in treatment of the highest contrast between blood and heart tissues in atrial fbrillation, 3D transesophageal echocardiography the model. Te MCC values help to obtain better recog- (3D-TEE) can be utilized to guide cardiac electrophysi- nition and segmentation efects when blood and myocar- ological intervention (locating ablation points) in real dial-echo contrast is low [48]. A variety of methods used time. However, there is a prerequisite that the changes for accuracy improvement of the endocardial boundary in the atrial anatomical structure have to be real-time Zhou et al. Cardiovasc Ultrasound (2021) 19:29 Page 7 of 11

monitored for precise positioning [49]. Accordingly, 3D leafet area, anterior and posterior leafet angle, non- Alexander et al. have applied an active shape model planar angle, prolapse, valve height, and volume. Previ- derived from computed tomography angiography (CTA) ous research has confrmed that there is no signifcant in a multi-chamber to segment the atrioventricle in a diference between automatic and manual measurements 3D-TEE image, improving segmentation accuracy in the [14]. Tis type of semi-automatic software has a favorable left atrium, which previously had a poor segmentation diagnostic value for valve morphology-related diseases performance. Furthermore, because 3D-TEE segmen- such as mitral valve prolapses. It also enhances the non- tation in the left atrium is vulnerable to scanning range experts’ accuracy in prolapse detection [35]. It can also restrictions, the team adopted fusion-imaging technol- be employed for intervalvular monitoring with advan- ogy to merge the CTA and 3D-TEE images for construc- tages of consistency and repeatability [36]. tion of wide-view 3D-TEE images, which contributed to Structural abnormalities in the congenital aortic valve, left atrial segmentation resolution and enhanced seg- senile aortic valve calcifcation, and rheumatic fever are mentation efciency. Tis research has an important sig- the common causes of aortic stenosis (AS). Te method nifcance for , where it can help to provide of aortic is crucial for severe AS. precise positioning for atrial fbrillation ablation [50, 51]. Before valve replacement surgery, valve diameter meas- Although AI technology can efectively segment atrio- urement and areas in 2D echocardiography have been ventricular structures in echocardiography, recognition manually detected to evaluate the level of valve stenosis and segmentation precision needs to be advanced fur- and provide a sufcient basis to determine the size of arti- ther due to interference of lung gas, ribs, and artifacts. fcial valves. However, because of the dynamic changes Algorithm updates, such as multiple iterations, efcient in the position of aortic valves in vivo, 2D static image- search methods, particle flters, and online collaborative based assessment is not only subjective but also only training approaches, combined with deep learning algo- reveals the measurement results of one or two frames rithms and multiple dynamic models [52] will further during the cardiac cycle. Some research studies [37] have optimize echocardiographic segmentation performance. adopted an automatic tracking algorithm based on ana- tomical afne optical fow for fast and automatic tracking Cardiac disease diagnosis with assistance of AI technology of aortic valves and proximal end of the left ventricular Valvular heart disease outfow tract using 3D-TEE technology to optimize the Routine echocardiography can visually investigate car- measurement. It provides dynamic and accurate support- diac valve shapes and activities. Its repeatability is not ing information for preoperative planning of aortic valve reliable due to subtle valve structure variations and wide replacement surgery, helping to improve the accuracy of heart motion ranges, as well as the intra- and inter- valve evaluation and enhancing surgeon confdence. observer diferences in recognizing valve stenosis, pro- lapse, calcifcation, and valve insufciency. Te mitral Cardiomyopathy and aortic valves are the objectives of AI technology in Cardiomyopathy can be classifed into two major catego- cardiac valve evaluation, especially focusing on observa- ries of primary and secondary cardiomyopathy according tion of valve morphology and regurgitation. Similarly, AI to the cause. AI-assisted echocardiographic diagnosis of technique is helpful in the assessment of valvular heart cardiomyopathy can be carried out via multiple imaging disease. For example, automatic evaluation software for modes, such as 2D ultrasound, M-mode, and color Dop- proximal isovelocity surface area (PISA) of mitral insuf- pler. Based on accurate wall recognition and ventricular ciency can conduct an automatic measurement of mitral segmentation, automatic measurement of ventricular valve regurgitant orifce area and regurgitant volume to volume measurement, accurate assessment of cardiac evaluate the severity of valve regurgitation. PISA for 3D function, and accurate visualization of myocardial move- echocardiography has a superior accuracy compared to ment through speckle tracking technology can be real- 2D image analysis software, which has excellent consist- ized. Te acquisition of these indicators helps to achieve ency with transesophageal ultrasound and MRI meas- rapid and precise detection of hypertrophic cardiomyo- urements [32, 33]. Another study [34] used real-time pathy and cardiac amyloidosis. In the latest research, a 3D volume color-fow Doppler technique to quantify human-interpretation-free machine learning pipeline valve regurgitation volume, which also achieved high based on the combination of ECG and echocardiography consistency in MRI measurements. In addition, valve had been developed to detect cardiac amyloidosis. Multi- morphology can be automatically analyzed by imple- center study had confrmed that the artifcial intelligence- menting automated measurements of the morphologi- enabled fully automated detection model outperformed cal mitral valve parameters, including 3D ring length interpretation by expert cardiologists in the diagnosis and height, 2D area, commissural width, overlap width, of cardiac amyloidosis [38]. On the other hand, speckle Zhou et al. Cardiovasc Ultrasound (2021) 19:29 Page 8 of 11

tracking imaging technology is widely used for the diag- heart disease combined with AI technology, experts nosis of cardiomyopathy. Myocardial strain analysis is have utilized the method of discrete wavelet transform helpful for diagnosis of various types of cardiomyopathy and texture feature analysis [42], showing that the AI [39]. Sengupta et al. created a machine learning algo- method using classifers to automatically sort echocar- rithm based on speckle tracking technology images to diographic images can provide the characteristic param- distinguish between constrictive pericarditis and restric- eters required for clinical diagnosis, as well as reduce the tive cardiomyopathy using an associative memory clas- occurrence of complications. In terms of 3D ultrasound sifer (AMC). As an additional challenge, both of the heart imaging, studies [43] have constructed myocar- diseases have similar ultrasonographic features, such as dial infarction models in pigs and sheep and applied the enlarged left and right atrium, relatively small ventricle, EchoPAC PC (GE, Andover, MA, USA) program to ana- pericardial efusion, and widening vena cava. Clinical lyze results. Tis method is a semi-automatic assessment diagnosis emphasizes the changes in myocardial strain of left ventricular quality and local strain values. In addi- parameters. For example, strain values of left ventricular tion to traditional ultrasound examinations, myocardial walls in constrictive pericarditis are signifcantly lower perfusion can diagnose by quan- than those of ventricular septa, while restrictive car- tifying the infarct area via examination of myocardial diomyopathy does not have this feature. Te study used contrast ultrasound. Te AI method can also perform an AMC to demonstrate that analysis of the frst 15 speckles automatic calculation of myocardial perfusion parame- tracking echocardiographic variables can classify diseases ters and reduce human error [15]. Echocardiography can more accurately (area under the curve (AUC) of 89.2%). be integrated with machine learning to obtain a huge set Tis method is better than the Doppler method alone of clinical data on echocardiographic changes. Machine (AUC of 82.1%) and the overall longitudinal strain (AUC learning can integrate these data into categories to assist of 63.7%), which obtained precision results. Te study physicians in accurately and rapidly diagnosing myo- also used machine learning to automatically distinguish cardial ischemia changes. In the prognosis of coronary between male patients with hypertrophic cardiomyopa- heart disease, a previous research had develop a method thy and athlete cardiac physiological hypertrophy [40]. based on texture parameters of native echocardiogram or Due to the diversifcation of ventricular morphology in contrast-enhanced acquisition to evaluate left ventricular patients with dilated cardiomyopathy (DCM), segmen- function recovery 1 year after myocardial infarction. Te tation of the left ventricular boundary (especially seg- highest rates of accurate prediction reached to 79% [44]. mentation near the ventricle apex) is more complex than that of the normal left ventricle. To address the ventricu- lar wall changes caused by DCM, Mahmood et al. [41] Diagnosis of intracardiac masses applied a support vector machine classifer to distinguish In addition to the auxiliary diagnosis of the above- between normal and dilated left ventricles. Although mentioned conventional diseases, AI technology can be the average classifcation accuracy of the study was only applied to classify and recognize intracardiac masses (like 77.8% (afected by cumulative errors), size evaluation left atrium/ear thrombosis, cardiac tumors and vegeta- accuracy of the left ventricle reached 87.2% and left ven- tion) [45]. For example, due to the complex and diverse tricular boundary segmentation accuracy was 89.3%. Te anatomical structures of the left atrial appendage and the ROI region recognition accuracy was 92.5%, providing a limited range of motion of the TEE probe, the resulting research foundation for further development of reliable ultrasound artifacts may lead to misdiagnosis. Diagno- decision-making tools. sis through images of the left atrial (auricular) thrombo- sis depend on the patient’s anticoagulant drug plan. Sun Coronary atherosclerotic heart disease et al. [46] performed transesophageal echocardiography Coronary heart disease, also known as coronary ath- on 130 patients with atrial fbrillation with image recon- erosclerotic heart disease, is one of the most common struction. Subsequently, the study extracted the texture coronary artery diseases, which is classifed as a special features of the gray-level co-occurrence image matrix and type of cardiomyopathy. Echocardiography can assist performed the classifcation using artifcial neural net- in its diagnosis by visually observing myocardial move- works. Te model classifcation AUC was 0.932, which is ment and changes in cardiac morphology. AI technol- much higher than sonographer’s diagnosis, where AUC ogy can efectively improve subjectivity of conventional was 0.834. Te study demonstrated that the Artifcial echocardiographic examination of coronary heart dis- Neural Network model can signifcantly improve TEE ease, increase detection systematicity, and help to better diagnosis of the left atrium (ear) thrombosis in patients distinguish between normal and infarcted myocardial with atrial fbrillation, which is conducive to early diag- images [53]. For the ultrasound diagnosis of coronary nosis and treatment [54]. Zhou et al. Cardiovasc Ultrasound (2021) 19:29 Page 9 of 11

Clinical application limitations of article and approval of article. All authors read and approved the fnal manuscript. In addition to black box efects, overftting problems, ethical issues, and other common limitations associated Funding with AI application in the medical feld [55], AI echocar- This work is supported by National Key R&D Program of China (2019YFE0110400). diography limitations also exist. First, echocardiography is not a bloodless mechanical operation. Communication Availability of data and materials between physicians and patient plays an indispensable Data sharing is not applicable to this article as no datasets were generated or analysed during the current study. role in disease diagnosis, while the current AI technology has not been able to perform human–computer interac- Declarations tions. In the future, there is a possibility that clinical work performed by a physician will be gradually substituted by Ethics approval and consent to participate AI, which will not in favor of further model optimization Not applicable. and sustainable development [56]. Based on our experi- Consent for publication ences, echocardiography and AI-integrated application, Not applicable. standardized data collection, and image annotation are Competing interests essential. Due to its complexities and rapid heartbeat, the The authors declare that they have no competing interests. echocardiographic section standardization is more dif- fcult compared to other organ section standardization, Author details 1 The First Afliated Hospital, Medical Imaging Centre, Hengyang Medical which may not conducive to the data collection in multi- School, University of South China, 69 Chuanshan Road, Hengyang 421001, center studies [57]. Furthermore, the diference of physi- China. 2 Institute of Medical Imaging, University of South China, Hengyang, ological and anatomical structures of the heart among China. races can not be ignore. It means that the model with a Received: 26 February 2021 Accepted: 5 August 2021 high accuracy on one specifc dataset may not be feasible on another data set [58].Terefore, while accepting the assistance of this technology, sonographers should also consider how to utilize its best functions and determine References what outcomes need to be adapted for patients. 1. Kuehn BM. on the cusp of an artifcial intelligence revolution. Circulation. 2020;141(15):1266–7. 2. Hannun AY, et al. Cardiologist- level arrhythmia detection and classif- cation in ambulatory electrocardiograms using a deep neural network. Conclusion and future outlook Nat Med. 2019;25:65–9. 3. Brandt V, Emrich T, Schoepf UJ, et al. Ischemia and outcome prediction Rich potential information contained in echocardio- by cardiac CT based machine learning. Int J Cardiovasc Imaging. 2020. grams can be extracted and utilized by AI technology https://​doi.​org/​10.​1007/​s10554-​020-​01929-y. [published online ahead to boost the accuracy of diagnosis. At present, AI tech- of print, 2020 Jul 4]. 4. Slomka PJ, Miller RJ, Isgum I, Dey D. Application and translation of nology software is used to examine cardiac ultrasounds, artifcial intelligence to cardiovascular imaging in nuclear medicine such that the process no longer depends as much on the and noncontrast CT. Semin Nucl Med. 2020;50(4):357–66. sonographer’s skills and experience, and with the added 5. Sanders WE Jr, Burton T, Khosousi A, et al. Machine learning: at the heart of failure diagnosis. Curr Opin Cardiol. 2021;36(2):227–33. beneft of rapid disease diagnosis. Te promising expan- 6. Kumar S, Nilsen WJ, Abernethy A, et al. Mobile health technology eval- sion of telemedicine with mobile and wireless technolo- uation: the health evidence workshop. Am J Prev Med. 2013;45:228–36. gies has produced unprecedented progress in the feld of 7. Chu WK, Raeside DE. Fourier analysis of the echocardiogram. Phys Med Biol. 1978;23(1):100–5. cardiovascular imaging [59]. With the implementation 8. Narang A, Bae R, Hong H, et al. Utility of a deep-learning algorithm to of remote echocardiography and application of robotic guide novices to acquire echocardiograms for limited diagnostic use. arms in 2014, as well as the realization of robot-assisted JAMA Cardiol. 2021. [Epub ahead of print]. 9. Alsharqi M, Woodward WJ, Mumith JA, et al. Artifcial intelligence and minimally invasive cardiac surgery, AI robotics technol- echocardiography. Echo Res Pract. 2018;5(4):115–25. ogy also provides sufcient support to acquire, identify, 10. Zhang J, Gajjala S, Agrawal P, et al. Fully automated echocardiogram and quantitatively analyze echocardiograms [60, 61]. interpretation in clinical practice. Circulation. 2018;138:1623–35. 11. Alsharqi M, Upton R, Mumith A, et al. Artifcial intelligence: a new clini- With the continued development of AI technology, the cal support tool for stress echocardiography. Expert Rev Med Devices. future pattern of echocardiography will gradually trans- 2018;15(8):513–5. form to provide practical auxiliary assistance for patients. 12. Madani A, Arnaout R, Mofrad M, et al. Fast and accurate view clas- sifcation of echocardiograms using deep learning. NPJ Digital Med. Acknowledgements 2018;1:6. Not applicable. 13. Zhang Y, Gao Y, Jiao J, et al. Robust boundary detection and tracking of left ventricles on ultrasound images using active shape model and ant Authors’ contributions colony optimization. Biomed Mater Eng. 2014;24(6):2893–9. Jia Zhou: Concept and drafting article. Meng Du: Inerpretation and drafting article. Shuai Chang: Critical revision of article. Zhiyi Chen: Critical revision Zhou et al. Cardiovasc Ultrasound (2021) 19:29 Page 10 of 11

14. Gandhi S, Mosleh W, Shen J, et al. Automation, machine learning, and arti- 34. Choi J, Hong GR, Kim M, et al. Automatic quantifcation of aortic regurgi- fcial intelligence in echocardiography: a brave new world. Echocardiog- tation using 3D full volume color : a validation raphy. 2018;35(9):1402–18. study with cardiac magnetic resonance imaging. Int J Cardiovasc Imag- 15. Li Y, Chahal N, Senior R, et al. Reproducible computer-assisted quanti- ing. 2015;31:1379–89. fcation of myocardial perfusion with contrast-enhanced ultrasound. 35. Kagiyama N, Toki M, Hara M, et al. Efcacy and accuracy of novel auto- Ultrasound Med Biol. 2017;43(10):2235–46. mated mitral valve quantifcation: three-dimensional transesophageal 16. Johnson KW, Jessica TS, Glicksberg BS, et al. Artifcial intelligence in cardi- echocardiographic study. Echocardiography. 2016;33:756–63. ology. J Am Coll Cardiol. 2018;71(23):2668–79. 36. Jin CN, Salgo IS, Schneider RJ, et al. Using anatomic intelligence to local- 17. Krittanawong C, Zhang H, Wang Z, Aydar M, Kitai T. Artifcial intel- ize mitral valve prolapse on three-dimensional echocardiography. J Am ligence in precision cardiovascular medicine. J Am Coll Cardiol. Soc Echocardiogr. 2016;29:938–45. 2017;69(21):2657–64. 37. Queirós S, Morais P, Fehske W, et al. Assessment of aortic valve tract 18. Litjens G, Ciompi F, Wolterink JM, et al. State-of-the-art deep learning in dynamics using automatic tracking of 3D transesophageal echocardio- cardiovascular image analysis. JACC Cardiovasc Imaging. 2019;12(8 Pt graphic images. Int J Cardiovasc Imaging. 2019;35(5):881–95. 1):1549–65. 38. Goto S, Mahara K, Beussink-Nelson L, et al. Artifcial intelligence-enabled 19. Østvik A, Smistad E, Aase SA, et al. Real-time standard view classifcation fully automated detection of cardiac amyloidosis using electrocardio- in transthoracic echocardiography using convolutional neural networks. grams and echocardiograms. Nat Commun. 2021;12(1):2726. Ultrasound Med Biol. 2019;45(2):374–84. 39. Kusunose K, Haga A, Abe T, et al. Utilization of artifcial intelligence in 20. Human vs AI-based echocardiography analysis as predictor of mortality echocardiography. Circ J. 2019;83(8):1623–9. in acute COVID-19 patients: WASE-COVID study. ACC Scientifc Sessions, 40. Sengupta PP, Huang YM, Bansal M, et al. Cognitive machine-learning 2021. algorithm for cardiac imaging: a pilot study for diferentiating constrictive 21. Li Y, Garson CD, Xu Y, Helm PA, Hossack JA, French BA. Serial ultrasound pericarditis from restrictive cardiomyopathy. Circ Cardiovasc Imaging. evaluation of intramyocardial strain after reperfused myocardial infarction 2016;9:e004330. reveals that remote zone dyssynchrony develops in concert with left 41. Mahmood R, Syeda-Mahmood T. Automatic detection of dilated cardio- ventricular remodeling. Ultrasound Med Biol. 2011;37(7):1073–86. myopathy in cardiac ultrasound videos. In: AMIA Annu Symp Proc. 2014. 22. de Alexandria A, Cortez P, Bessa J, et al. pSnakes: a new radial active con- p. 865–871. tour model and its application in the segmentation of the left ventricle 42. Kusunose K, Abe T, Haga A, et al. A deep learning approach for assess- from echocardiographic images. Comput Methods Programs Biomed. ment of regional wall motion abnormality from echocardiographic 2014;116(3):260–73. images. JACC Cardiovasc Imaging. 2019;S1936–878X(19):30318–3. 23. Qin X, Cong Z, Fei B, et al. Automatic segmentation of right ventricular 43. Streif C, Zhu M, Panosian J, et al. Comprehensive evaluation of cardiac ultrasound images using sparse matrix transform and a level set. Phys function and detection of myocardial infarction based on a semi-auto- Med Biol. 2013;8(21):7609–24. mated analysis using full-volume real time three-dimensional echocardi- 24. Bersvendsen J, Orderud F, Lie Ø, et al. Semiautomated biventricular ography. Echocardiography. 2015;32(2):332–8. segmentation in three-dimensional echocardiography by coupled 44. Strzelecki M, Skonieczka S, Kasprzak JD, et al. Analysis of myocardial tex- deformable surfaces. J Med Imaging. 2017;4(2):024005. ture in resting echocardiographic images predicts recovery 1 year after 25. Tsang W, Salgo IS, Medvedofsky D, et al. Transthoracic 3D Echocardio- myocardial infarction. In: 2016 Signal Processing: Algorithms, Architec- graphic Left Heart Chamber Quantifcation Using an Automated Adap- tures, Arrangements, and Applications (SPA) IEEE. 2016. tive Analytics Algorithm. JACC Cardiovasc Imaging. 2016;9(7):769–82. 45. Strzelecki M, Materka A, Drozdz J, et al. Classifcation and segmentation 26. Tamborini G, Piazzese C, Lang RM, et al. Feasibility and accuracy of of intracardiac masses in cardiac tumor echocardiograms. Comput Med automated software for transthoracic three-dimensional left ventricu- Imaging Graph. 2006;30(2):95–107. lar volume and function analysis: comparisons with two-dimensional 46. Sun L, Li Y, Zhang YT, et al. A computer-aided diagnostic algorithm echocardiography, three-dimensional transthoracic manual method, improves the accuracy of transesophageal echocardiography for left and cardiac magnetic resonance imaging. J Am Soc Echocardiogr. atrial thrombi. J Ultrasound Med. 2014;33(1):83–91. 2017;30(11):1049–58. 47. Schneider M, Bartko P, Geller W, et al. A machine learning algorithm 27. Knackstedt C, Bekkers SCAM, Schummers G, et al. Fully automated supports ultrasound-naïve novices in the acquisition of diagnostic versus standard tracking of left ventricular ejection fraction and echocardiography loops and provides accurate estimation of LVEF. Int J longitudinal strain: the FAST-EFs multicenter study. J Am Coll Cardiol. Cardiovasc Imaging. 2021;37(2):577–86. 2015;66(13):1456–66. 48. Saris AE, Nillesen MM, Lopata RG, et al. Correlation-based discrimina- 28. Asch FM, Poilvert N, Abraham T, et al. Automated echocardiographic tion between cardiac tissue and blood for segmentation of the left quantifcation of left ventricular ejection fraction without volume ventricle in 3-D echocardiographic images. Ultrasound Med Biol. measurements using a machine learning algorithm mimicking a human 2014;40(3):596–610. expert. Circ Cardiovasc Imaging. 2019;12(9):e009303. 49. Chin CG, Chung FP, Lin YJ, et al. The application of novel segmentation 29. Kusunose K, Haga A, Yamaguchi N, et al. Deep learning for assessment software to create left atrial geometry for atrial fbrillation ablation: the of left ventricular ejection fraction from echocardiographic images. J Am implication of spatial resolution. J Chin Med Assoc. [published online Soc Echocardiogr. 2020;33(5):632-635.e1. ahead of print]. 30. Salte IM, Østvik A, Smistad E, et al. Artifcial intelligence for automatic 50. Alexander H, Ben R, Harriet W, et al. Improved segmentation of multiple measurement of left ventricular strain in echocardiography. JACC Cardio- cavities of the heart in wideview 3-D transesophageal echocardiograms. vasc Imaging. 2021;S1936–878X(21)00363–6. https://​doi.​org/​10.​1016/j.​ Ultrasound Med Biol. 2015;41(7):1991–2000. jcmg.​2021.​04.​018. 51. Haak A, Vegas-Sánchez-Ferrero G, Mulder H, et al. Segmentation of mul- 31. Asch FM, Mor-Avi V, Rubenson D, et al. Deep learning-based automated tiple heart cavities in 3-D transesophageal ultrasound images. IEEE Trans echocardiographic quantifcation of left ventricular ejection fraction: a Ultrason Ferroelectr Freq Control. 2015;62(6):1179–89. point-of-care solution. Circ Cardiovasc Imaging. 2021;14(6):e012293. 52. Carneiro G, Nascimento JC. Multiple dynamic models for tracking the 32. Thavendiranathan P, Liu S, Datta S, et al. Quantifcation of chronic func- left ventricle of the heart from ultrasound data using particle flters and tional mitral regurgitation by automated 3-dimensional peak and inte- deep learning architectures. In: 2010 IEEE Computer Society Conference grated proximal isovelocity surface area and stroke volume techniques on Computer Vision and Pattern Recognition. IEEE; 2010. p. 2815–2822. using real-time 3-dimensional volume color Doppler echocardiography: https://​ieeex​plore.​ieee.​org/​docum​ent/​55400​13. in vitro and clinical validation. Circ Cardiovasc Imaging. 2013;6:125–33. 53. Sudarshan V, Acharya UR, Ng EY, et al. Automated identifcation of 33. de Agustin JA, Marcos-Alberca P, Fernandez-Golfn C, et al. Direct infarcted myocardium tissue characterization using ultrasound images: a measurement of proximal isovelocity surface area by singlebeat three- review. IEEE Rev Biomed Eng. 2015;8:86–97. dimensional color Doppler echocardiography in mitral regurgitation: a 54. Liu CX, Jiao D, Liu Z. Artifcial Intelligence (AI)-aided disease prediction. validation study. J Am Soc Echocardiogr. 2012;25:815–23. BIO Integration. 2020;1(3):130–6. Zhou et al. Cardiovasc Ultrasound (2021) 19:29 Page 11 of 11

55. Krittanawong C, Johnson KW, Rosenson RS, et al. Deep learning for car- 60. Arbeille P, Provost R, Zuj K, et al. Teles-operated echocardiography diovascular medicine: a practical primer. Eur Heart J. 2019;40(25):2058–73. using a robotic arm and an internet connection. Ultrasound Med Biol. 56. Dey D, Slomka PJ, Leeson P, et al. Artifcial intelligence in cardiovascular 2014;40(10):2521–9. imaging. J Am Coll Cardiol. 2019;73(11):1317–35. 61. Michael E, Jonathan T, Grace E, et al. Transesophageal echocardiography 57. Nabi W, Bansal A, Xu B. Applications of artifcial intelligence and guidance for robot-assisted level III inferior vena cava tumor thrombec- machine learning approaches in echocardiography. Echocardiography. tomy: a novel approach to intraoperative care. J Cardiothorac Vasc 2021;38(6):982–92. Anesth. 2018;32:S1053077018303495. 58. Ye Z, Kumar Y, Sing G, et al. Deep echocardiography: a frst step toward automatic cardiac disease diagnosis using machine learning. J Internet Technol. 2020;21(6):1589–600. Publisher’s Note 59. Seetharam K, Kagiyama N, Sengupta PP. Application of mobile health, Springer Nature remains neutral with regard to jurisdictional claims in pub- telemedicine and artifcial intelligence to echocardiography. Echo Res lished maps and institutional afliations. Pract. 2019;6(2):R41–52.

Ready to submit your research ? Choose BMC and benefit from:

• fast, convenient online submission • thorough peer review by experienced researchers in your field • rapid publication on acceptance • support for research data, including large and complex data types • gold Open Access which fosters wider collaboration and increased citations • maximum visibility for your research: over 100M website views per year

At BMC, research is always in progress.

Learn more biomedcentral.com/submissions