VW<RXWK$FDGHPLF$QQXDO&RQIHUHQFHRI&KLQHVH$VVRFLDWLRQRI$XWRPDWLRQ :XKDQ&KLQD1RYHPEHU 2YHUYLHZRI'HHS/HDUQLQJ ;XHGDQ'X <LQJKDR&DL 6KXR:DQJDQG/HLMLH=KDQJ 7KH6WDWH.H\/DERUDWRU\RI0DQDJHPHQWDQG&RQWUROIRU&RPSOH[6\VWHPV ,QVWLWXWHRI$XWRPDWLRQ&$6 %HLMLQJ&KLQD (PDLOV GX[XHGDQ#LDDFFQ\LQJKDRFDL#LDDFFQVKXRZDQJ#LDDFFQ]KDQJOHLMLH#LDDFFQ Abstract²,Q UHFHQW \HDUV GHHS OHDUQLQJ KDV DFKLHYHG JUHDW SHUIRUPDQFH RI WUDGLWLRQDO PDFKLQH OHDUQLQJ PHWKRGV XVXDOO\ VXFFHVV LQ PDQ\ ILHOGV VXFK DV FRPSXWHU YLVLRQ DQG QDWXUDO UHO\RQXVHUV¶H[SHULHQFHVZKLOH GHHSOHDUQLQJDSSURDFKHVUHO\ ODQJXDJH SURFHVVLQJ &RPSDUHG WR WUDGLWLRQDO PDFKLQH OHDUQLQJ RQ WKH GDWD 7KHUHIRUH ZH FDQ ILQG RXW WKDW GHHS OHDUQLQJ PHWKRGVGHHSOHDUQLQJKDVDVWURQJOHDUQLQJDELOLW\DQGFDQPDNH DSSURDFKHV KDYH UHGXFHG WKH GHPDQGV IRU XVHUV :LWK WKH EHWWHU XVH RI GDWDVHWV IRU IHDWXUH H[WUDFWLRQ %HFDXVH RI LWV SURJUHVV RI FRPSXWHU WHFKQRORJ\ FRPSXWHUV¶ SHUIRUPDQFH LV SUDFWLFDELOLW\GHHSOHDUQLQJEHFRPHVPRUHDQGPRUHSRSXODUIRU UDSLGO\LPSURYHG0HDQZKLOHLQIRUPDWLRQRQWKH,QWHUQHWLVDOVR PDQ\UHVHDUFKHUVWRGRUHVHDUFKZRUNV,QWKLVSDSHUZHPDLQO\ VSHZLQJRXW7KHVHIDFWRUVSURYLGHDVWURQJLPSHWXVIRUGHHS LQWURGXFHVRPHDGYDQFHGQHXUDOQHWZRUNVRIGHHSOHDUQLQJDQG OHDUQLQJ WR GHYHORS DQG PDNH GHHS OHDUQLQJ EHFRPH WKH WKHLU DSSOLFDWLRQV %HVLGHV ZH DOVR GLVFXVV WKH OLPLWDWLRQV DQG SUHYDOHQWPHWKRGLQPDFKLQHOHDUQLQJ SURVSHFWVRIGHHSOHDUQLQJ ,QWKLV SDSHU ZH PDNHDV\VWHPDWLFLQWURGXFWLRQIRUGHHS Keywords—deep learning; machine learning; neural network OHDUQLQJ IURP PDQ\ DVSHFWV E\ H[SDWLDWLQJ LWV UHVHDUFK SURJUHVVHV VWDWHRIWKHDUW PRGHOV IUDPHZRUNV DQG , ,1752'8&7,21 DSSOLFDWLRQV UHVSHFWLYHO\ )LUVW ZH LQWURGXFH WKH UHVHDUFK 'HHSOHDUQLQJZDVGHYHORSHGIURPDUWLILFLDOQHXUDOQHWZRUN SURJUHVVHVLQ6HFWLRQ,,7KHQZHLQWURGXFHVHYHUDOW\SLFDOGHHS DQGQRZLWLVDSUHYDOHQWILHOGRIPDFKLQHOHDUQLQJ7KHUHVHDUFK OHDUQLQJ PRGHOV LQ 6HFWLRQ ,,, DQG VHYHUDO GHHS OHDUQLQJ RIDUWLILFLDOQHXUDOQHWZRUNEHJDQIURPV0F&XOORFK et al IUDPHZRUNVLQ6HFWLRQ,91H[WZHOLVWVRPHDSSOLFDWLRQVRI >@SURSRVHGWKH0F&XOORFK3LWWV 03 PRGHOE\DQDO\]LQJDQG GHHSOHDUQLQJLQ6HFWLRQ9)LQDOO\ZHFRQFOXGHWKLVSDSHULQ VXPPDUL]LQJ WKH FKDUDFWHULVWLFV RI QHXURQV +HEE et al >@ 6HFWLRQ9, SURSRVHG D FHOO DVVHPEO\ WKHRU\ WR H[SODLQ WKH DGDSWDWLRQ RI FHUHEUDOQHXURQGXULQJWKHOHDUQLQJSURFHVV7KLVWKHRU\KDGDQ ,, 5(6($5&+352*5(66(6 LPSRUWDQW LQIOXHQFH RQ WKH GHYHORSPHQW RI QHXUDO QHWZRUNV 7KHFRQFHSWRIGHHSOHDUQLQJZDVSXWIRUZDUGLQDWILUVW 7KHQ 5RVHQEODWW et al >@ LQYHQWHG WKH SHUFHSWURQ DOJRULWKP $IWHUWKDWGHHSOHDUQLQJLVVWLOOFRQWLQXDOO\GHYHORSLQJDWDEURDG 7KLVDOJRULWKPLVDNLQGRIELQDU\FODVVLILHUZKLFKEHORQJVWR $WSUHVHQWWKHUHDUHPDQ\RXWVWDQGLQJILJXUHVVXFKDV*HRIIUH\ VXSHUYLVHG OHDUQLQJ :LGURZ SURSRVHG WKH DGDSWLYH OLQHDU +LQWRQ<RVKXD%HQJLR<DQQ/H&XQDQG$QGUHZ1J7KH\DUH HOHPHQWDQGLWLVDVLQJOHOD\HUDUWLILFLDOQHXUDOQHWZRUNEDVHG OHDGLQJWKHUHVHDUFKGLUHFWLRQRIGHHSOHDUQLQJ6RPHFRPSDQLHV RQWKH03PRGHO8QIRUWXQDWHO\0LQVN\DQG3DSHUWSRLQWHGWKDW OLNH *RRJOH DQG )DFHERRN KDYH PDGH ORWV RI UHVHDUFK WKH SHUFHSWURQ DOJRULWKP KDG JUHDW OLPLWDWLRQV LQ WKHRU\ DQG DFKLHYHPHQWVLQGHHSOHDUQLQJDQGDSSOLHGWKHPWRYDULRXVILHOGV PDGHDQHJDWLYHHYDOXDWLRQRQWKHSURVSHFWVRIQHXUDOQHWZRUNV ,QWKLV\HDU*RRJOH¶V$OSKD*RSURJUDPGHIHDWHG/HH6HGROLQ ZKLFK OHG WKH GHYHORSPHQW RI QHXUDO QHWZRUNV WR KLW D QDGLU *RFRPSHWLWLRQZKLFKVKRZHGWKDWGHHSOHDUQLQJKDGDVWURQJ +RZHYHU+RSILHOGet al>@SURSRVHGWKH+RSILHOGQHWZRUNLQ OHDUQLQJ DELOLW\ :KDW¶V PRUH *RRJOH¶V 'HHS'UHDP LV DQ WKH HDUO\ V 7KLV PDGH DUWLILFLDO QHXUDO QHWZRUN UHYLYHG H[FHOOHQW VRIWZDUH ZKLFK FDQ QRW RQO\ FODVVLI\ LPDJHV EXW 7KHQ+LQWRQet al. >@SURSRVHGWKH%ROW]PDQQPDFKLQHE\XVLQJ JHQHUDWH VWUDQJH DQG DUWLILFLDO SDLQWLQJV EDVHG RQ LWV RZQ VLPXODWHG DQQHDOLQJ DOJRULWKP ,Q WKH V YDULRXV VKDOORZ NQRZOHGJH )DFHERRN DQQRXQFHG D QHZ DUWLILFLDO LQWHOOLJHQFH PDFKLQHOHDUQLQJPHWKRGVZHUHSURSRVHGRQHDIWHUDQRWKHUVXFK V\VWHPQDPHG'HHS7H[W'HHS7H[WLVDGHHSOHDUQLQJEDVHG DV VXSSRUW YHFWRU PDFKLQH >@ %RRVWLQJ >@ 'XH WR WKH WH[WXQGHUVWDQGLQJHQJLQHZKLFKFDQFODVVLI\PDVVLYHDPRXQWV DGYDQWDJHVRIWKHVHPHWKRGVERWKLQWKHRU\DQGLQDSSOLFDWLRQ RIGDWDSURYLGHFRUUHVSRQGLQJVHUYLFHVDIWHULGHQWLI\LQJXVHUV¶ DUWLILFLDOQHXUDOQHWZRUNKLWDQDGLUDJDLQ$IWHU+LQWRQHWDOSXW FKDWWLQJPHVVDJHVDQGFOHDQXSVSDPPHVVDJHV IRUZDUGWKHFRQFHSWRIGHHSOHDUQLQJLQWKHMRXUQDO6FLHQFHLQ DUWLILFLDOQHXUDOQHWZRUNRQFHDJDLQUHFHLYHGPXFKLQWHUHVW 'HHS OHDUQLQJ VWDUWHG UHODWLYHO\ ODWH EXW GHYHORSHG YHU\ IURPWKHUHVHDUFKFRPPXQLW\ UDSLGO\ DW KRPH 7KHUH KDYH DFKLHYHG UHPDUNDEOH SURJUHVV LQ FROOHJHVXQLYHUVLWLHVUHVHDUFKLQVWLWXWHVDQGFRPSDQLHV%DLGX 'HHSOHDUQLQJPRGHOVXVXDOO\DGRSWKLHUDUFKLFDOVWUXFWXUHV KDV HVWDEOLVKHG D GHHS OHDUQLQJ LQVWLWXWH WR H[SORUH KRZ WR WRFRQQHFWWKHLUOD\HUV7KHRXWSXWRIDORZHUOD\HUFDQEH FRPSOHWH PDQ\ D WDVN ZLWK GHHS OHDUQLQJ %DLGX¶V XQPDQQHG UHJDUGHG DV WKH LQSXW RI D KLJKHU OD\HU YLD VLPSOH OLQHDU RU JURXQGYHKLFOHKDVDFFRPSOLVKHGURDGWHVWXQGHUFRPSOLFDWHG QRQOLQHDU FDOFXODWLRQV 7KHVH PRGHOV FDQ WUDQVIRUP ORZOHYHO URDG FRQGLWLRQV ,)/<7(. VWDUWHG WKH UHVHDUFK RI VSHHFK IHDWXUHVRIWKHGDWDLQWR KLJKOHYHODEVWUDFWIHDWXUHV2ZQLQJ WR UHFRJQLWLRQ EDVHG RQ 'HHS 1HXUDO 1HWZRUN '11 LQ WKLV FKDUDFWHULVWLF GHHS OHDUQLQJ PRGHOV FDQ EH VWURQJHU WKDQ 7KH\ ODXQFKHG WKH ILUVW RQOLQH &KLQHVH VSHHFK UHFRJQLWLRQ VKDOORZPDFKLQHOHDUQLQJPRGHOVLQIHDWXUHUHSUHVHQWDWLRQ7KH V\VWHP DQG DQ DGYDQFHG WHFKQRORJ\ WR UHFRJQL]H GLIIHUHQW 7KLVZRUNLVVXSSRUWHGLQSDUWE\WKH1DWLRQDO1DWXUDO6FLHQFH )RXQGDWLRQ RI&KLQDXQGHU*UDQWV ,((( ODQJXDJHV$QGQRZWKH\KDYHSXEOLVKHGDKLJKSHUIRUPDQFH FRPSXWLQJ +3& SODWIRUPLQFRRSHUDWLRQZLWK,QWHO \ / [[ÿ ,,, '((3/($51,1*02'(/6 )URPWKHEHJLQQLQJWRWKHSUHVHQWWKHUHDUHDORWRIGHHS OHDUQLQJPRGHOV 7KHW\SLFDOPRGHOVLQFOXGH$XWRHQFRGHU $( 'HHS %HOLHI 1HWZRUN '%1 &RQYROXWLRQDO 1HXUDO 1HWZRUN &11 DQG5HFXUUHQW1HXUDO1HWZRUN 511 ,QWKLVVHFWLRQ ZHPDLQO\LQWURGXFHVRPHVWDWHRIWKHDUWPRGHOV [ [ÿ A. Autoencoder )LJ 7KHEDVLFSULQFLSOHRI$XWRHQFRGHU $XWRHQFRGHU LV PDLQO\ XVHG WR SURFHVV FRPSOH[ KLJK GLPHQVLRQDOGDWD,WVDLPLVWROHDUQKRZWRUHSUHVHQWDVHWRIGDWD +LGGHQOD\HUh YLDGLPHQVLRQDOLW\UHGXFWLRQ:KHQZHSURFHVVWKHLQSXWx E\ XVLQJDVHULHVRIZHLJKWLQJDQGPDSSLQJPHWKRGVZHFDQJHWWKH ORZGLPHQVLRQDO RXWSXWy7KHQZHDGRSWWKHLQYHUVHZHLJKWLQJ DQG PDSSLQJ PHWKRGV WR PDNH y WUDQVIRUP WR WKH RXWSXW x’ ZKRVHGLPHQVLRQLVDVWKHVDPHDVWKHLQSXWx1RZDOOZHKDYH 9LVXDOOD\HUv WRGRLVWRPDNHWKHHUURUIXQFWLRQL x, x’ EHWKHVPDOOHVWE\ WUDLQLQJLWHUDWLYHO\WKHQHWZRUNZHLJKWV7KHEDVLFSULQFLSOHRI )LJ 7KHEDVLFVWUXFWXUHRIUHVWULFWHG%ROW]PDQQPDFKLQH $(LVVKRZQLQ)LJ )LJ $( DOVR KDV PDQ\ LPSURYHG VWUXFWXUHV OLNH 'HQRLVLQJ $XWRHQFRGHU >@ DQG 6SDUVH $XWRHQFRGHU >@ )RU 'HQRLVLQJ $IWHULQFUHDVLQJWKHQXPEHURIWKHKLGGHQOD\HUVRI5%0 $XWRHQFRGHULWXVHVWKHRULJLQDOGDWDZLWKUDQGRPQRLVHWRWUDLQ ZHFDQJHWGHHS%ROW]PDQQPDFKLQH7KHQZHDGRSWDWRSGRZQ QHWZRUNZHLJKWVZKLFKPDNHVH[WUDFWHGIHDWXUHVEHFRPHPRUH GLUHFWHGFRQQHFWLRQQHDUWKHYLVXDOOD\HUVRWKDWZHFDQJHW'%1 UREXVW)RU6SDUVH$XWRHQFRGHUEHVLGHVLQFUHDVLQJWKHQXPEHU PRGHO :KHQ WUDLQLQJ WKH QHWZRUN WKH JUHHG\ XQVXSHUYLVHG RI KLGGHQ OD\HUV DQG QHXURQV 6SDUVH $XWRHQFRGHU OLPLWV WKH OD\HUZLVHSUHWUDLQLQJPHWKRGFDQEHXVHGWRJHWWKHQHWZRUN DFWLYDWLRQVWDWHRIKLGGHQQRGHVZKLFKRQO\DVPDOOQXPEHURI ZHLJKWV,WRQO\WUDLQVRQHOD\HUDWDWLPHZLWKWKHRXWSXWRIWKH KLGGHQQRGHVDUHLQWKHDFWLYDWHGVWDWHDQGPRVWRIKLGGHQQRGHV ORZHUOD\HUEHLQJXVHGDVWKHLQSXWRIWKHKLJKHUOD\HU7KHQ DUH LQWKHXQDFWLYDWHGVWDWH EDFNSURSDJDWLRQ DOJRULWKP LV XVHG WR ILQHWXQH WKH ZKROH QHWZRUN ;LRQJet al>@SURSRVHGDPRGLILHGDXWRHQFRGHUQHWZRUN WR UHFRJQL]H DQG VHSDUDWH DQRPDORXV RQHV IURP D VHW RI /LX et al >@ SURSRVHG D QRYHO %RRVWHG 'HHS %HOLHI JHRFKHPLFDO VDPSOHV &RQWLQXRXV 5HVWULFWHG %ROW]PDQQ 1HWZRUN %'%1 ZKLFKFRQVLVWVRIVHYHUDO'%1V(DFK'%1 0DFKLQH &5%0 LVXVHGDVWKHSDUWRIWKHDXWRHQFRGHUQHWZRUN LVXVHGWROHDUQKLHUDUFKLFDOIHDWXUHUHSUHVHQWDWLRQVDQGDOO'%1V LQ>@7KHDXWKRUVDGRSWWKUHHVWHSVWRWUDLQWKHPRGHOZKLFK ZKLFK DUH UHJDUGHG DV ZHDN OHDUQHUV DUH FRQQHFWHG WRJHWKHU DUH SUHWUDLQLQJ &5%0V XQUROOLQJ &5%1V WR FRQVWUXFW WKH WKURXJK D ERRVWHG FODVVLILHU %'%1 DGRSWV D ERWWRPXS QHWZRUN DQG ILQHWXQLQJ SDUDPHWHUV YLD EDFNSURSDJDWLRQ XQVXSHUYLVHGIHDWXUHOHDUQLQJ %88)/ SURFHVVDQGDERRVWHG )LQDOO\ WKLV DSSURDFK DFKLHYHV JRRG UHVXOWV LQ UHFRJQL]LQJ WRSGRZQVXSHUYLVHGIHDWXUHVWUHQJWKHQ %7'6)6 SURFHVV,W PXOWLYDULDWHJHRFKHPLFDODQRPDOLHV LV XVHG WR UHFRJQL]H IDFLDO H[SUHVVLRQV 7KH QHWZRUN GLYLGHV IDFLDOLPDJHVLQWRPDQ\SDUWLDOO\RYHUODSSHGSDWFKHV$QGWKHQ /RXL]RV et al >@ SURSRVHG D YDULDWLRQDO IDLU DXWRHQFRGHU LW XVHV %88)/ SURFHVV WR OHDUQ IHDWXUH UHSUHVHQWDWLRQ IURP PRGHO ZKLFK FRXOG PDNH ODWHQW UHSUHVHQWDWLRQV PD[LPDOO\ HDFKSDWFKZLWKRQH'%1DQG%7'6)6SURFHVVWRILQHWXQHWKH LQIRUPDWLYH DERXW REVHUYHG UDQGRP YDULDEOHV EXW PLQLPDOO\ IHDWXUHVE\SURFHVVLQJFODVVLILFDWLRQHUURUVSURGXFHGE\ERRVWHG LQIRUPDWLYHDERXWVHQVLWLYHRUQXLVDQFHYDULDEOHV,QRWKHUZRUGV FODVVLILHUDQGZHDNOHDUQHUV)LQDOO\WKHPRGHOJHWVEHWWHUUHVXOWV WKHPRGHOFRXOGVHSDUDWHXQGHVLUHGIDFWRUVIURPWKHYDULDWLRQV WKDQRWKHUUHODWHGZRUN7KHVWUXFWXUHRIWKLVPRGHOLVVKRZQLQ ZKLOH UHWDLQLQJ DV PXFK LQIRUPDWLRQ DV SRVVLEOH IURP ZKDW )LJ UHPDLQV,QRUGHUWRUHPRYHVHQVLWLYHRUQXLVDQFHYDULDEOHVIURP ODWHQW UHSUHVHQWDWLRQV >@ DGGHG D SHQDOW\ WHUP EDVHG RQ .LP et al >@ SURSRVHG D ILQJHUSULQW OLYHQHVV GHWHFWLRQ 0D[LPXP 0HDQ 'LVFUHSDQF\ PHDVXUH WR WKH PRGHO $W ODVW PHWKRGZKLFKFRXOGGLVWLQJXLVKZKHWKHUDVFDQQHGILQJHUSULQWLV WKH\DSSOLHGWKLVPRGHOWRVRPHWDVNVDQGJRWJUHDWUHVXOWV OLYHRUIDNHSULRUWRWKHUHFRJQLWLRQ7KHPRGHOLQ>@XVHVD GHHSEHOLHIQHWZRUN7KHVWUXFWXUHRI'%1LVDVWKHVDPHDVD B. Deep Belief Network QRUPDO '%1 H[FHSW IRU WKH ODVW OD\HU ZKLFK KDV WZR RXWSXW 'HHS%HOLHI1HWZRUNLVDNLQGRIQHXUDOQHWZRUNZKLFKLV QRGHVWRPDNHOLYHQHVVGHFLVLRQ%HIRUHLQSXWWLQJGDWDWR'%1 VWDFNHG E\ VHYHUDO UHVWULFWHG %ROW]PDQQ PDFKLQHV 5%0V ILQJHUSULQWLPDJHVVKRXOGEHSURFHVVHG7KHDXWKRUVXVHWKHWZR 5%0LVDNLQGRIJHQHUDWLYHVWRFKDVWLFQHXUDOQHWZRUNPRGHOV GLPHQVLRQDO+DUULVFRUQHUGHWHFWRUWRLQIHUWKHDYHUDJHORFDWLRQ ZKLFKFRPHVIURPWKH%ROW]PDQQPDFKLQH$OWKRXJK5%0KDV ZKLFKFRQWDLQHGWKHUHJLRQRILQWHUHVW7KHPRGHOLVSUHWUDLQHG