DOCSLIB.ORG

Overview of Deep Learning

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Overview of Deep Learning VW<RXWK$FDGHPLF$QQXDO&RQIHUHQFHRI&KLQHVH$VVRFLDWLRQRI$XWRPDWLRQ :XKDQ&KLQD1RYHPEHU 2YHUYLHZRI'HHS/HDUQLQJ ;XHGDQ'X <LQJKDR&DL 6KXR:DQJDQG/HLMLH=KDQJ 7KH6WDWH.H\/DERUDWRU\RI0DQDJHPHQWDQG&RQWUROIRU&RPSOH[6\VWHPV ,QVWLWXWHRI$XWRPDWLRQ&$6 %HLMLQJ&KLQD (PDLOV GX[XHGDQ#LDDFFQ\LQJKDRFDL#LDDFFQVKXRZDQJ#LDDFFQ]KDQJOHLMLH#LDDFFQ Abstract²,Q UHFHQW \HDUV GHHS OHDUQLQJ KDV DFKLHYHG JUHDW SHUIRUPDQFH RI WUDGLWLRQDO PDFKLQH OHDUQLQJ PHWKRGV XVXDOO\ VXFFHVV LQ PDQ\ ILHOGV VXFK DV FRPSXWHU YLVLRQ DQG QDWXUDO UHO\RQXVHUV¶H[SHULHQFHVZKLOH GHHSOHDUQLQJDSSURDFKHVUHO\ ODQJXDJH SURFHVVLQJ &RPSDUHG WR WUDGLWLRQDO PDFKLQH OHDUQLQJ RQ WKH GDWD 7KHUHIRUH ZH FDQ ILQG RXW WKDW GHHS OHDUQLQJ PHWKRGVGHHSOHDUQLQJKDVDVWURQJOHDUQLQJDELOLW\DQGFDQPDNH DSSURDFKHV KDYH UHGXFHG WKH GHPDQGV IRU XVHUV :LWK WKH EHWWHU XVH RI GDWDVHWV IRU IHDWXUH H[WUDFWLRQ %HFDXVH RI LWV SURJUHVV RI FRPSXWHU WHFKQRORJ\ FRPSXWHUV¶ SHUIRUPDQFH LV SUDFWLFDELOLW\GHHSOHDUQLQJEHFRPHVPRUHDQGPRUHSRSXODUIRU UDSLGO\LPSURYHG0HDQZKLOHLQIRUPDWLRQRQWKH,QWHUQHWLVDOVR PDQ\UHVHDUFKHUVWRGRUHVHDUFKZRUNV,QWKLVSDSHUZHPDLQO\ VSHZLQJRXW7KHVHIDFWRUVSURYLGHDVWURQJLPSHWXVIRUGHHS LQWURGXFHVRPHDGYDQFHGQHXUDOQHWZRUNVRIGHHSOHDUQLQJDQG OHDUQLQJ WR GHYHORS DQG PDNH GHHS OHDUQLQJ EHFRPH WKH WKHLU DSSOLFDWLRQV %HVLGHV ZH DOVR GLVFXVV WKH OLPLWDWLRQV DQG SUHYDOHQWPHWKRGLQPDFKLQHOHDUQLQJ SURVSHFWVRIGHHSOHDUQLQJ ,QWKLV SDSHU ZH PDNHDV\VWHPDWLFLQWURGXFWLRQIRUGHHS Keywords—deep learning; machine learning; neural network OHDUQLQJ IURP PDQ\ DVSHFWV E\ H[SDWLDWLQJ LWV UHVHDUFK SURJUHVVHV VWDWHRIWKHDUW PRGHOV IUDPHZRUNV DQG , ,1752'8&7,21 DSSOLFDWLRQV UHVSHFWLYHO\ )LUVW ZH LQWURGXFH WKH UHVHDUFK 'HHSOHDUQLQJZDVGHYHORSHGIURPDUWLILFLDOQHXUDOQHWZRUN SURJUHVVHVLQ6HFWLRQ,,7KHQZHLQWURGXFHVHYHUDOW\SLFDOGHHS DQGQRZLWLVDSUHYDOHQWILHOGRIPDFKLQHOHDUQLQJ7KHUHVHDUFK OHDUQLQJ PRGHOV LQ 6HFWLRQ ,,, DQG VHYHUDO GHHS OHDUQLQJ RIDUWLILFLDOQHXUDOQHWZRUNEHJDQIURPV0F&XOORFK et al IUDPHZRUNVLQ6HFWLRQ,91H[WZHOLVWVRPHDSSOLFDWLRQVRI >@SURSRVHGWKH0F&XOORFK3LWWV 03 PRGHOE\DQDO\]LQJDQG GHHSOHDUQLQJLQ6HFWLRQ9)LQDOO\ZHFRQFOXGHWKLVSDSHULQ VXPPDUL]LQJ WKH FKDUDFWHULVWLFV RI QHXURQV +HEE et al >@ 6HFWLRQ9, SURSRVHG D FHOO DVVHPEO\ WKHRU\ WR H[SODLQ WKH DGDSWDWLRQ RI FHUHEUDOQHXURQGXULQJWKHOHDUQLQJSURFHVV7KLVWKHRU\KDGDQ ,, 5(6($5&+352*5(66(6 LPSRUWDQW LQIOXHQFH RQ WKH GHYHORSPHQW RI QHXUDO QHWZRUNV 7KHFRQFHSWRIGHHSOHDUQLQJZDVSXWIRUZDUGLQDWILUVW 7KHQ 5RVHQEODWW et al >@ LQYHQWHG WKH SHUFHSWURQ DOJRULWKP $IWHUWKDWGHHSOHDUQLQJLVVWLOOFRQWLQXDOO\GHYHORSLQJDWDEURDG 7KLVDOJRULWKPLVDNLQGRIELQDU\FODVVLILHUZKLFKEHORQJVWR $WSUHVHQWWKHUHDUHPDQ\RXWVWDQGLQJILJXUHVVXFKDV*HRIIUH\ VXSHUYLVHG OHDUQLQJ :LGURZ SURSRVHG WKH DGDSWLYH OLQHDU +LQWRQ<RVKXD%HQJLR<DQQ/H&XQDQG$QGUHZ1J7KH\DUH HOHPHQWDQGLWLVDVLQJOHOD\HUDUWLILFLDOQHXUDOQHWZRUNEDVHG OHDGLQJWKHUHVHDUFKGLUHFWLRQRIGHHSOHDUQLQJ6RPHFRPSDQLHV RQWKH03PRGHO8QIRUWXQDWHO\0LQVN\DQG3DSHUWSRLQWHGWKDW OLNH *RRJOH DQG )DFHERRN KDYH PDGH ORWV RI UHVHDUFK WKH SHUFHSWURQ DOJRULWKP KDG JUHDW OLPLWDWLRQV LQ WKHRU\ DQG DFKLHYHPHQWVLQGHHSOHDUQLQJDQGDSSOLHGWKHPWRYDULRXVILHOGV PDGHDQHJDWLYHHYDOXDWLRQRQWKHSURVSHFWVRIQHXUDOQHWZRUNV ,QWKLV\HDU*RRJOH¶V$OSKD*RSURJUDPGHIHDWHG/HH6HGROLQ ZKLFK OHG WKH GHYHORSPHQW RI QHXUDO QHWZRUNV WR KLW D QDGLU *RFRPSHWLWLRQZKLFKVKRZHGWKDWGHHSOHDUQLQJKDGDVWURQJ +RZHYHU+RSILHOGet al>@SURSRVHGWKH+RSILHOGQHWZRUNLQ OHDUQLQJ DELOLW\ :KDW¶V PRUH *RRJOH¶V 'HHS'UHDP LV DQ WKH HDUO\ V 7KLV PDGH DUWLILFLDO QHXUDO QHWZRUN UHYLYHG H[FHOOHQW VRIWZDUH ZKLFK FDQ QRW RQO\ FODVVLI\ LPDJHV EXW 7KHQ+LQWRQet al. >@SURSRVHGWKH%ROW]PDQQPDFKLQHE\XVLQJ JHQHUDWH VWUDQJH DQG DUWLILFLDO SDLQWLQJV EDVHG RQ LWV RZQ VLPXODWHG DQQHDOLQJ DOJRULWKP ,Q WKH V YDULRXV VKDOORZ NQRZOHGJH )DFHERRN DQQRXQFHG D QHZ DUWLILFLDO LQWHOOLJHQFH PDFKLQHOHDUQLQJPHWKRGVZHUHSURSRVHGRQHDIWHUDQRWKHUVXFK V\VWHPQDPHG'HHS7H[W'HHS7H[WLVDGHHSOHDUQLQJEDVHG DV VXSSRUW YHFWRU PDFKLQH >@ %RRVWLQJ >@ 'XH WR WKH WH[WXQGHUVWDQGLQJHQJLQHZKLFKFDQFODVVLI\PDVVLYHDPRXQWV DGYDQWDJHVRIWKHVHPHWKRGVERWKLQWKHRU\DQGLQDSSOLFDWLRQ RIGDWDSURYLGHFRUUHVSRQGLQJVHUYLFHVDIWHULGHQWLI\LQJXVHUV¶ DUWLILFLDOQHXUDOQHWZRUNKLWDQDGLUDJDLQ$IWHU+LQWRQHWDOSXW FKDWWLQJPHVVDJHVDQGFOHDQXSVSDPPHVVDJHV IRUZDUGWKHFRQFHSWRIGHHSOHDUQLQJLQWKHMRXUQDO6FLHQFHLQ DUWLILFLDOQHXUDOQHWZRUNRQFHDJDLQUHFHLYHGPXFKLQWHUHVW 'HHS OHDUQLQJ VWDUWHG UHODWLYHO\ ODWH EXW GHYHORSHG YHU\ IURPWKHUHVHDUFKFRPPXQLW\ UDSLGO\ DW KRPH 7KHUH KDYH DFKLHYHG UHPDUNDEOH SURJUHVV LQ FROOHJHVXQLYHUVLWLHVUHVHDUFKLQVWLWXWHVDQGFRPSDQLHV%DLGX 'HHSOHDUQLQJPRGHOVXVXDOO\DGRSWKLHUDUFKLFDOVWUXFWXUHV KDV HVWDEOLVKHG D GHHS OHDUQLQJ LQVWLWXWH WR H[SORUH KRZ WR WRFRQQHFWWKHLUOD\HUV7KHRXWSXWRIDORZHUOD\HUFDQEH FRPSOHWH PDQ\ D WDVN ZLWK GHHS OHDUQLQJ %DLGX¶V XQPDQQHG UHJDUGHG DV WKH LQSXW RI D KLJKHU OD\HU YLD VLPSOH OLQHDU RU JURXQGYHKLFOHKDVDFFRPSOLVKHGURDGWHVWXQGHUFRPSOLFDWHG QRQOLQHDU FDOFXODWLRQV 7KHVH PRGHOV FDQ WUDQVIRUP ORZOHYHO URDG FRQGLWLRQV ,)/<7(. VWDUWHG WKH UHVHDUFK RI VSHHFK IHDWXUHVRIWKHGDWDLQWR KLJKOHYHODEVWUDFWIHDWXUHV2ZQLQJ WR UHFRJQLWLRQ EDVHG RQ 'HHS 1HXUDO 1HWZRUN '11 LQ WKLV FKDUDFWHULVWLF GHHS OHDUQLQJ PRGHOV FDQ EH VWURQJHU WKDQ 7KH\ ODXQFKHG WKH ILUVW RQOLQH &KLQHVH VSHHFK UHFRJQLWLRQ VKDOORZPDFKLQHOHDUQLQJPRGHOVLQIHDWXUHUHSUHVHQWDWLRQ7KH V\VWHP DQG DQ DGYDQFHG WHFKQRORJ\ WR UHFRJQL]H GLIIHUHQW 7KLVZRUNLVVXSSRUWHGLQSDUWE\WKH1DWLRQDO1DWXUDO6FLHQFH )RXQGDWLRQ RI&KLQDXQGHU*UDQWV ,((( ODQJXDJHV$QGQRZWKH\KDYHSXEOLVKHGDKLJKSHUIRUPDQFH FRPSXWLQJ +3& SODWIRUPLQFRRSHUDWLRQZLWK,QWHO \ / [[ÿ ,,, '((3/($51,1*02'(/6 )URPWKHEHJLQQLQJWRWKHSUHVHQWWKHUHDUHDORWRIGHHS OHDUQLQJPRGHOV 7KHW\SLFDOPRGHOVLQFOXGH$XWRHQFRGHU $( 'HHS %HOLHI 1HWZRUN '%1 &RQYROXWLRQDO 1HXUDO 1HWZRUN &11 DQG5HFXUUHQW1HXUDO1HWZRUN 511 ,QWKLVVHFWLRQ ZHPDLQO\LQWURGXFHVRPHVWDWHRIWKHDUWPRGHOV [ [ÿ A. Autoencoder )LJ 7KHEDVLFSULQFLSOHRI$XWRHQFRGHU $XWRHQFRGHU LV PDLQO\ XVHG WR SURFHVV FRPSOH[ KLJK GLPHQVLRQDOGDWD,WVDLPLVWROHDUQKRZWRUHSUHVHQWDVHWRIGDWD +LGGHQOD\HUh YLDGLPHQVLRQDOLW\UHGXFWLRQ:KHQZHSURFHVVWKHLQSXWx E\ XVLQJDVHULHVRIZHLJKWLQJDQGPDSSLQJPHWKRGVZHFDQJHWWKH ORZGLPHQVLRQDO RXWSXWy7KHQZHDGRSWWKHLQYHUVHZHLJKWLQJ DQG PDSSLQJ PHWKRGV WR PDNH y WUDQVIRUP WR WKH RXWSXW x’ ZKRVHGLPHQVLRQLVDVWKHVDPHDVWKHLQSXWx1RZDOOZHKDYH 9LVXDOOD\HUv WRGRLVWRPDNHWKHHUURUIXQFWLRQL x, x’ EHWKHVPDOOHVWE\ WUDLQLQJLWHUDWLYHO\WKHQHWZRUNZHLJKWV7KHEDVLFSULQFLSOHRI )LJ 7KHEDVLFVWUXFWXUHRIUHVWULFWHG%ROW]PDQQPDFKLQH $(LVVKRZQLQ)LJ )LJ $( DOVR KDV PDQ\ LPSURYHG VWUXFWXUHV OLNH 'HQRLVLQJ $XWRHQFRGHU >@ DQG 6SDUVH $XWRHQFRGHU >@ )RU 'HQRLVLQJ $IWHULQFUHDVLQJWKHQXPEHURIWKHKLGGHQOD\HUVRI5%0 $XWRHQFRGHULWXVHVWKHRULJLQDOGDWDZLWKUDQGRPQRLVHWRWUDLQ ZHFDQJHWGHHS%ROW]PDQQPDFKLQH7KHQZHDGRSWDWRSGRZQ QHWZRUNZHLJKWVZKLFKPDNHVH[WUDFWHGIHDWXUHVEHFRPHPRUH GLUHFWHGFRQQHFWLRQQHDUWKHYLVXDOOD\HUVRWKDWZHFDQJHW'%1 UREXVW)RU6SDUVH$XWRHQFRGHUEHVLGHVLQFUHDVLQJWKHQXPEHU PRGHO :KHQ WUDLQLQJ WKH QHWZRUN WKH JUHHG\ XQVXSHUYLVHG RI KLGGHQ OD\HUV DQG QHXURQV 6SDUVH $XWRHQFRGHU OLPLWV WKH OD\HUZLVHSUHWUDLQLQJPHWKRGFDQEHXVHGWRJHWWKHQHWZRUN DFWLYDWLRQVWDWHRIKLGGHQQRGHVZKLFKRQO\DVPDOOQXPEHURI ZHLJKWV,WRQO\WUDLQVRQHOD\HUDWDWLPHZLWKWKHRXWSXWRIWKH KLGGHQQRGHVDUHLQWKHDFWLYDWHGVWDWHDQGPRVWRIKLGGHQQRGHV ORZHUOD\HUEHLQJXVHGDVWKHLQSXWRIWKHKLJKHUOD\HU7KHQ DUH LQWKHXQDFWLYDWHGVWDWH EDFNSURSDJDWLRQ DOJRULWKP LV XVHG WR ILQHWXQH WKH ZKROH QHWZRUN ;LRQJet al>@SURSRVHGDPRGLILHGDXWRHQFRGHUQHWZRUN WR UHFRJQL]H DQG VHSDUDWH DQRPDORXV RQHV IURP D VHW RI /LX et al >@ SURSRVHG D QRYHO %RRVWHG 'HHS %HOLHI JHRFKHPLFDO VDPSOHV &RQWLQXRXV 5HVWULFWHG %ROW]PDQQ 1HWZRUN %'%1 ZKLFKFRQVLVWVRIVHYHUDO'%1V(DFK'%1 0DFKLQH &5%0 LVXVHGDVWKHSDUWRIWKHDXWRHQFRGHUQHWZRUN LVXVHGWROHDUQKLHUDUFKLFDOIHDWXUHUHSUHVHQWDWLRQVDQGDOO'%1V LQ>@7KHDXWKRUVDGRSWWKUHHVWHSVWRWUDLQWKHPRGHOZKLFK ZKLFK DUH UHJDUGHG DV ZHDN OHDUQHUV DUH FRQQHFWHG WRJHWKHU DUH SUHWUDLQLQJ &5%0V XQUROOLQJ &5%1V WR FRQVWUXFW WKH WKURXJK D ERRVWHG FODVVLILHU %'%1 DGRSWV D ERWWRPXS QHWZRUN DQG ILQHWXQLQJ SDUDPHWHUV YLD EDFNSURSDJDWLRQ XQVXSHUYLVHGIHDWXUHOHDUQLQJ %88)/ SURFHVVDQGDERRVWHG )LQDOO\ WKLV DSSURDFK DFKLHYHV JRRG UHVXOWV LQ UHFRJQL]LQJ WRSGRZQVXSHUYLVHGIHDWXUHVWUHQJWKHQ %7'6)6 SURFHVV,W PXOWLYDULDWHJHRFKHPLFDODQRPDOLHV LV XVHG WR UHFRJQL]H IDFLDO H[SUHVVLRQV 7KH QHWZRUN GLYLGHV IDFLDOLPDJHVLQWRPDQ\SDUWLDOO\RYHUODSSHGSDWFKHV$QGWKHQ /RXL]RV et al >@ SURSRVHG D YDULDWLRQDO IDLU DXWRHQFRGHU LW XVHV %88)/ SURFHVV WR OHDUQ IHDWXUH UHSUHVHQWDWLRQ IURP PRGHO ZKLFK FRXOG PDNH ODWHQW UHSUHVHQWDWLRQV PD[LPDOO\ HDFKSDWFKZLWKRQH'%1DQG%7'6)6SURFHVVWRILQHWXQHWKH LQIRUPDWLYH DERXW REVHUYHG UDQGRP YDULDEOHV EXW PLQLPDOO\ IHDWXUHVE\SURFHVVLQJFODVVLILFDWLRQHUURUVSURGXFHGE\ERRVWHG LQIRUPDWLYHDERXWVHQVLWLYHRUQXLVDQFHYDULDEOHV,QRWKHUZRUGV FODVVLILHUDQGZHDNOHDUQHUV)LQDOO\WKHPRGHOJHWVEHWWHUUHVXOWV WKHPRGHOFRXOGVHSDUDWHXQGHVLUHGIDFWRUVIURPWKHYDULDWLRQV WKDQRWKHUUHODWHGZRUN7KHVWUXFWXUHRIWKLVPRGHOLVVKRZQLQ ZKLOH UHWDLQLQJ DV PXFK LQIRUPDWLRQ DV SRVVLEOH IURP ZKDW )LJ UHPDLQV,QRUGHUWRUHPRYHVHQVLWLYHRUQXLVDQFHYDULDEOHVIURP ODWHQW UHSUHVHQWDWLRQV >@ DGGHG D SHQDOW\ WHUP EDVHG RQ .LP et al >@ SURSRVHG D ILQJHUSULQW OLYHQHVV GHWHFWLRQ 0D[LPXP 0HDQ 'LVFUHSDQF\ PHDVXUH WR WKH PRGHO $W ODVW PHWKRGZKLFKFRXOGGLVWLQJXLVKZKHWKHUDVFDQQHGILQJHUSULQWLV WKH\DSSOLHGWKLVPRGHOWRVRPHWDVNVDQGJRWJUHDWUHVXOWV OLYHRUIDNHSULRUWRWKHUHFRJQLWLRQ7KHPRGHOLQ>@XVHVD GHHSEHOLHIQHWZRUN7KHVWUXFWXUHRI'%1LVDVWKHVDPHDVD B. Deep Belief Network QRUPDO '%1 H[FHSW IRU WKH ODVW OD\HU ZKLFK KDV WZR RXWSXW 'HHS%HOLHI1HWZRUNLVDNLQGRIQHXUDOQHWZRUNZKLFKLV QRGHVWRPDNHOLYHQHVVGHFLVLRQ%HIRUHLQSXWWLQJGDWDWR'%1 VWDFNHG E\ VHYHUDO UHVWULFWHG %ROW]PDQQ PDFKLQHV 5%0V ILQJHUSULQWLPDJHVVKRXOGEHSURFHVVHG7KHDXWKRUVXVHWKHWZR 5%0LVDNLQGRIJHQHUDWLYHVWRFKDVWLFQHXUDOQHWZRUNPRGHOV GLPHQVLRQDO+DUULVFRUQHUGHWHFWRUWRLQIHUWKHDYHUDJHORFDWLRQ ZKLFKFRPHVIURPWKH%ROW]PDQQPDFKLQH$OWKRXJK5%0KDV ZKLFKFRQWDLQHGWKHUHJLRQRILQWHUHVW7KHPRGHOLVSUHWUDLQHG
Load more

Recommended publications
  • Deep Belief Networks for Phone Recognition
    Deep Belief Networks for phone recognition Abdel-rahman Mohamed, George Dahl, and Geoffrey Hinton Department of Computer Science University of Toronto {asamir,gdahl,hinton}@cs.toronto.edu Abstract Hidden Markov Models (HMMs) have been the state-of-the-art techniques for acoustic modeling despite their unrealistic independence assumptions and the very limited representational capacity of their hidden states. There are many proposals in the research community for deeper models that are capable of modeling the many types of variability present in the speech generation process. Deep Belief Networks (DBNs) have recently proved to be very effective for a variety of ma- chine learning problems and this paper applies DBNs to acoustic modeling. On the standard TIMIT corpus, DBNs consistently outperform other techniques and the best DBN achieves a phone error rate (PER) of 23.0% on the TIMIT core test set. 1 Introduction A state-of-the-art Automatic Speech Recognition (ASR) system typically uses Hidden Markov Mod- els (HMMs) to model the sequential structure of speech signals, with local spectral variability mod- eled using mixtures of Gaussian densities. HMMs make two main assumptions. The first assumption is that the hidden state sequence can be well-approximated using a first order Markov chain where each state St at time t depends only on St−1. Second, observations at different time steps are as- sumed to be conditionally independent given a state sequence. Although these assumptions are not realistic, they enable tractable decoding and learning even with large amounts of speech data. Many methods have been proposed for relaxing the very strong conditional independence assumptions of standard HMMs (e.g.
    [Show full text]
  • A Survey Paper on Deep Belief Network for Big Data
    International Journal of Computer Engineering & Technology (IJCET) Volume 9, Issue 5, September-October 2018, pp. 161–166, Article ID: IJCET_09_05_019 Available online at http://iaeme.com/Home/issue/IJCET?Volume=9&Issue=5 Journal Impact Factor (2016): 9.3590(Calculated by GISI) www.jifactor.com ISSN Print: 0976-6367 and ISSN Online: 0976–6375 © IAEME Publication A SURVEY PAPER ON DEEP BELIEF NETWORK FOR BIG DATA Azhagammal Alagarsamy Research Scholar, Bharathiyar University, Coimbatore, Tamil Nadu, India Assistant Professor, Department of Computer Science, Ayya Nadar Janaki Ammal College, Sivakasi, Tamilnadu Dr. K. Ruba Soundar Head, Department of CSE, PSR Engineering College, Sivakasi, Tamilnadu, India ABSTRACT Deep Belief Network (DBN) , as one of the deep learning architecture also noticeable machine learning technique, used in many applications like image processing, speech recognition and text mining. It uses supervised and unsupervised method to understand features in hierarchical architecture for the classification and pattern recognition. Recent year development in technologies has enabled the collection of large data. It also poses many challenges in data mining and processing due its nature of large volume, variety, velocity and veracity. In this paper we review the research of DBN for Big data. Key words: DBN, Big Data, Deep Belief Network, Classification. Cite this Article: Azhagammal Alagarsamy and Dr. K. Ruba Soundar, A Survey Paper on Deep Belief Network for Big Data. International Journal of Computer Engineering and Technology, 9(5), 2018, pp. 161-166. http://iaeme.com/Home/issue/IJCET?Volume=9&Issue=5 1. INTRODUCTION Deep learning is also named as deep structured learning or hierarchal learning.
    [Show full text]
  • Auto-Encoding a Knowledge Graph Using a Deep Belief Network
    ABSTRACT We started with a knowledge graph of connected entities and descriptive properties of those entities, from which, a hierarchical representation of the knowledge graph is derived. Using a graphical, energy-based neural network, we are able to show that the structure of the hierarchy can be internally captured by the neural network, which allows for efficient output of the underlying equilibrium distribution from which the data are drawn. AUTO-ENCODING A Robert A. Murphy [email protected] KNOWLEDGE GRAPH USING A DEEP BELIEF NETWORK A Random Fields Perspective Table of Contents Introduction .................................................................................................................................................. 2 GloVe for Knowledge Expansion ................................................................................................................... 2 The Approach ................................................................................................................................................ 3 Deep Belief Network ................................................................................................................................. 4 Random Field Setup .............................................................................................................................. 4 Random Field Illustration ...................................................................................................................... 5 Restricted Boltzmann Machine ................................................................................................................
    [Show full text]
  • Unsupervised Pre-Training of a Deep LSTM-Based Stacked Autoencoder for Multivariate Time Series Forecasting Problems Alaa Sagheer 1,2,3* & Mostafa Kotb2,3
    www.nature.com/scientificreports OPEN Unsupervised Pre-training of a Deep LSTM-based Stacked Autoencoder for Multivariate Time Series Forecasting Problems Alaa Sagheer 1,2,3* & Mostafa Kotb2,3 Currently, most real-world time series datasets are multivariate and are rich in dynamical information of the underlying system. Such datasets are attracting much attention; therefore, the need for accurate modelling of such high-dimensional datasets is increasing. Recently, the deep architecture of the recurrent neural network (RNN) and its variant long short-term memory (LSTM) have been proven to be more accurate than traditional statistical methods in modelling time series data. Despite the reported advantages of the deep LSTM model, its performance in modelling multivariate time series (MTS) data has not been satisfactory, particularly when attempting to process highly non-linear and long-interval MTS datasets. The reason is that the supervised learning approach initializes the neurons randomly in such recurrent networks, disabling the neurons that ultimately must properly learn the latent features of the correlated variables included in the MTS dataset. In this paper, we propose a pre-trained LSTM- based stacked autoencoder (LSTM-SAE) approach in an unsupervised learning fashion to replace the random weight initialization strategy adopted in deep LSTM recurrent networks. For evaluation purposes, two diferent case studies that include real-world datasets are investigated, where the performance of the proposed approach compares favourably with the deep LSTM approach. In addition, the proposed approach outperforms several reference models investigating the same case studies. Overall, the experimental results clearly show that the unsupervised pre-training approach improves the performance of deep LSTM and leads to better and faster convergence than other models.
    [Show full text]
  • A Video Recognition Method by Using Adaptive Structural Learning Of
    A Video Recognition Method by using Adaptive Structural Learning of Long Short Term Memory based Deep Belief Network Shin Kamada Takumi Ichimura Advanced Artificial Intelligence Project Research Center, Advanced Artificial Intelligence Project Research Center, Research Organization of Regional Oriented Studies, Research Organization of Regional Oriented Studies, Prefectural University of Hiroshima and Faculty of Management and Information System, 1-1-71, Ujina-Higashi, Minami-ku, Prefectural University of Hiroshima Hiroshima 734-8558, Japan 1-1-71, Ujina-Higashi, Minami-ku, E-mail: [email protected] Hiroshima 734-8558, Japan E-mail: [email protected] Abstract—Deep learning builds deep architectures such as function that classifies an given image or detects an object multi-layered artificial neural networks to effectively represent while predicting the future, is required. Understanding of time multiple features of input patterns. The adaptive structural series video is expected in various kinds industrial fields, learning method of Deep Belief Network (DBN) can realize a high classification capability while searching the optimal network such as human detection, pose or facial estimation from video structure during the training. The method can find the optimal camera, autonomous driving system, and so on [10]. number of hidden neurons of a Restricted Boltzmann Machine LSTM (Long Short Term Memory) is a well-known method (RBM) by neuron generation-annihilation algorithm to train the for time-series prediction and is applied to deep learning given input data, and then it can make a new layer in DBN methods[11]. The method enabled the traditional recurrent by the layer generation algorithm to actualize a deep data representation.
    [Show full text]
  • Deep Reinforcement Learning with Experience Replay Based on SARSA
    Deep Reinforcement Learning with Experience Replay Based on SARSA Dongbin Zhao, Haitao Wang, Kun Shao and Yuanheng Zhu Key Laboratory of Management and Control for Complex Systems Institute of Automation Chinese Academy of Sciences, Beijing 100190, China [email protected], [email protected], [email protected], [email protected] Abstract—SARSA, as one kind of on-policy reinforcement designed to deal with decision-making problems, it has run into learning methods, is integrated with deep learning to solve the great difficulties when handling high dimension data. With the video games control problems in this paper. We use deep development of feature detection method like DL, such convolutional neural network to estimate the state-action value, problems are to be well solved. and SARSA learning to update it. Besides, experience replay is introduced to make the training process suitable to scalable A new method, called deep reinforcement learning (DRL), machine learning problems. In this way, a new deep emerges to lead the direction of advanced AI research. DRL reinforcement learning method, called deep SARSA is proposed combines excellent perceiving ability of DL with decision- to solve complicated control problems such as imitating human making ability of RL. In 2010, Lange [10] proposed a typical to play video games. From the experiments results, we can algorithm which applied a deep auto-encoder neural network conclude that the deep SARSA learning shows better (DANN) into a visual control task. Later, Abtahi and Fasel [11] performances in some aspects than deep Q learning. employed a deep belief network (DBN) as the function approximation to improve the learning efficiency of traditional Keywords—SARSA learning; Q learning; experience replay; neural fitted-Q method.
    [Show full text]
  • Deep Belief Networks Based Feature Generation and Regression For
    Deep Belief Networks Based Feature Generation and Regression for Predicting Wind Power Asifullah Khan, Aneela Zameer*, Tauseef Jamal, Ahmad Raza Email Addresses: ([asif, aneelaz, jamal]@pieas.edu.pk, [email protected]) Department of Computer Science, Pakistan Institute of Engineering and Applied Sciences (PIEAS), Nilore, Islamabad, Pakistan. Aneela Zameer*: Email: [email protected]; [email protected] Phone: +92 3219799379 Fax: 092 519248600 Page 1 of 31 ABSTRACT Wind energy forecasting helps to manage power production, and hence, reduces energy cost. Deep Neural Networks (DNN) mimics hierarchical learning in the human brain and thus possesses hierarchical, distributed, and multi-task learning capabilities. Based on aforementioned characteristics, we report Deep Belief Network (DBN) based forecast engine for wind power prediction because of its good generalization and unsupervised pre-training attributes. The proposed DBN-WP forecast engine, which exhibits stochastic feature generation capabilities and is composed of multiple Restricted Boltzmann Machines, generates suitable features for wind power prediction using atmospheric properties as input. DBN-WP, due to its unsupervised pre-training of RBM layers and generalization capabilities, is able to learn the fluctuations in the meteorological properties and thus is able to perform effective mapping of the wind power. In the deep network, a regression layer is appended at the end to predict sort-term wind power. It is experimentally shown that the deep learning and unsupervised pre-training capabilities of DBN based model has comparable and in some cases better results than hybrid and complex learning techniques proposed for wind power prediction. The proposed prediction system based on DBN, achieves mean values of RMSE, MAE and SDE as 0.124, 0.083 and 0.122, respectively.
    [Show full text]
  • Deep Belief Networks
    Probabilistic Graphical Models Statistical and Algorithmic Foundations of Deep Learning Eric Xing Lecture 11, February 19, 2020 Reading: see class homepage © Eric Xing @ CMU, 2005-2020 1 ML vs DL © Eric Xing @ CMU, 2005-2020 2 Outline q An overview of DL components q Historical remarks: early days of neural networks q Modern building blocks: units, layers, activations functions, loss functions, etc. q Reverse-mode automatic differentiation (aka backpropagation) q Similarities and differences between GMs and NNs q Graphical models vs. computational graphs q Sigmoid Belief Networks as graphical models q Deep Belief Networks and Boltzmann Machines q Combining DL methods and GMs q Using outputs of NNs as inputs to GMs q GMs with potential functions represented by NNs q NNs with structured outputs q Bayesian Learning of NNs q Bayesian learning of NN parameters q Deep kernel learning © Eric Xing @ CMU, 2005-2020 3 Outline q An overview of DL components q Historical remarks: early days of neural networks q Modern building blocks: units, layers, activations functions, loss functions, etc. q Reverse-mode automatic differentiation (aka backpropagation) q Similarities and differences between GMs and NNs q Graphical models vs. computational graphs q Sigmoid Belief Networks as graphical models q Deep Belief Networks and Boltzmann Machines q Combining DL methods and GMs q Using outputs of NNs as inputs to GMs q GMs with potential functions represented by NNs q NNs with structured outputs q Bayesian Learning of NNs q Bayesian learning of NN parameters
    [Show full text]
  • Non-Destructive Detection of Tea Leaf Chlorophyll Content Using Hyperspectral Reflectance and Machine Learning Algorithms
    plants Article Non-Destructive Detection of Tea Leaf Chlorophyll Content Using Hyperspectral Reflectance and Machine Learning Algorithms Rei Sonobe 1,* , Yuhei Hirono 2 and Ayako Oi 2 1 Faculty of Agriculture, Shizuoka University, Shizuoka 422-8529, Japan 2 Institute of Fruit Tree and Tea Science, National Agriculture and Food Research Organization, Shimada 428-8501, Japan; hirono@affrc.go.jp (Y.H.); nayako@affrc.go.jp (A.O.) * Correspondence: [email protected]; Tel.: +81-54-2384839 Received: 24 December 2019; Accepted: 16 March 2020; Published: 17 March 2020 Abstract: Tea trees are kept in shaded locations to increase their chlorophyll content, which influences green tea quality. Therefore, monitoring change in chlorophyll content under low light conditions is important for managing tea trees and producing high-quality green tea. Hyperspectral remote sensing is one of the most frequently used methods for estimating chlorophyll content. Numerous studies based on data collected under relatively low-stress conditions and many hyperspectral indices and radiative transfer models show that shade-grown tea performs poorly. The performance of four machine learning algorithms—random forest, support vector machine, deep belief nets, and kernel-based extreme learning machine (KELM)—in evaluating data collected from tea leaves cultivated under different shade treatments was tested. KELM performed best with a root-mean-square error of 8.94 3.05 µg cm 2 and performance to deviation values from 1.70 to 8.04 for the test data. ± − These results suggest that a combination of hyperspectral reflectance and KELM has the potential to trace changes in the chlorophyll content of shaded tea leaves.
    [Show full text]
  • A Deep Belief Network Classification Approach for Automatic
    applied sciences Article A Deep Belief Network Classification Approach for Automatic Diacritization of Arabic Text Waref Almanaseer * , Mohammad Alshraideh and Omar Alkadi King Abdullah II School for Information Technology, The University of Jordan, Amman 11942, Jordan; [email protected] (M.A.); [email protected] (O.A.) * Correspondence: [email protected] Abstract: Deep learning has emerged as a new area of machine learning research. It is an approach that can learn features and hierarchical representation purely from data and has been successfully applied to several fields such as images, sounds, text and motion. The techniques developed from deep learning research have already been impacting the research on Natural Language Processing (NLP). Arabic diacritics are vital components of Arabic text that remove ambiguity from words and reinforce the meaning of the text. In this paper, a Deep Belief Network (DBN) is used as a diacritizer for Arabic text. DBN is an algorithm among deep learning that has recently proved to be very effective for a variety of machine learning problems. We evaluate the use of DBNs as classifiers in automatic Arabic text diacritization. The DBN was trained to individually classify each input letter with the corresponding diacritized version. Experiments were conducted using two benchmark datasets, the LDC ATB3 and Tashkeela. Our best settings achieve a DER and WER of 2.21% and 6.73%, receptively, on the ATB3 benchmark with an improvement of 26% over the best published results. On the Tashkeela benchmark, our system continues to achieve high accuracy with a DER of 1.79% and 14% improvement.
    [Show full text]
  • The Effects of Deep Belief Network Pre-Training of a Multilayered Perceptron Under Varied Labeled Data Conditions
    EXAMENSARBETE INOM TEKNIK, GRUNDNIVÅ, 15 HP STOCKHOLM, SVERIGE 2016 The effects of Deep Belief Network pre-training of a Multilayered perceptron under varied labeled data conditions CHRISTOFFER MÖCKELIND MARCUS LARSSON KTH SKOLAN FÖR DATAVETENSKAP OCH KOMMUNIKATION Royal Institute of Technology The effects of Deep Belief Network pre-training of a Multilayered perceptron under varied labeled data conditions Effekterna av att initialisera en MLP med en tränad DBN givet olika begränsningar av märkt data Author: Supervisor: Christoffer Möckelind Pawel Herman Examinator: Marcus Larsson Örjan Ekeberg May 11, 2016 Abstract Sometimes finding labeled data for machine learning tasks is difficult. This is a problem for purely supervised models like the Multilayered perceptron(MLP). ADiscriminativeDeepBeliefNetwork(DDBN)isasemi-supervisedmodelthat is able to use both labeled and unlabeled data. This research aimed to move towards a rule of thumb of when it is beneficial to use a DDBN instead of an MLP, given the proportions of labeled and unlabeled data. Several trials with different amount of labels, from the MNIST and Rectangles-Images datasets, were conducted to compare the two models. It was found that for these datasets, the DDBNs had better accuracy when few labels were available. With 50% or more labels available, the DDBNs and MLPs had comparable accuracies. It is concluded that a rule of thumb of using a DDBN when less than 50% of labels are available for training, would be in line with the results. However, more research is needed to make any general conclusions. Sammanfattning Märkt data kan ibland vara svårt att hitta för maskininlärningsuppgifter. Detta är ett problem för modeller som bygger på övervakad inlärning, exem- pelvis Multilayerd Perceptron(MLP).
    [Show full text]
  • A Deep Belief Network Approach to Learning Depth from Optical Flow
    A Deep Belief Network Approach to Learning Depth from Optical Flow Reuben Feinman Honors Thesis Division of Applied Mathematics, Brown University Providence, RI 02906 Advised by Thomas Serre and Stuart Geman Abstract It is well established that many living beings make use of motion information encoded in the visual system as a means to understand the surrounding environ- ment and to assist with navigation tasks. In particular, a phenomenon referred to as motion parallax is known to be instrumental in the perception of depth when binocular cues are distorted or otherwise unavailable [6]. The key idea is that objects which are closer to a given observer appear to move faster through the visual field than those at a further distance. Due to this phenomenon, optical flow can be a very powerful tool for the decoding of spatial information. When exact data is available regarding the speed and direction of optical flow, recovering depth is trivial [1]. In general, however, sources of this information are unreliable, therefore a learning approach can come of great use in this domain. We describe the application of a deep belief network (DBN) to the task of learn- ing depth from optical flow in videos. The network takes as input motion features generated by a hierarchical feedforward architecture of spatio-temporal feature de- tectors [4]. This system was developed as an extension of the HMAX model [12] and is inspired by the neurobiology of motion processing in the visual cortex. Our network is trained using a set of graphical scenes generated by a gaming engine, such that ground truth depth information is available for each video sequence.
    [Show full text]