NeurIPS 2020 Meetup Beijing
Controllable and Interpretable Machine Learning for Natural Language Generation
Lei Li ByteDance AI Lab
12/6/2020 Revolution in Information Creation and Sharing • New media platforms
• Tremendous improvement in the efficiency and quality of content creation • Massive distribution of personalized information
2 AI for Information Creation and Sharing
Information Creation & AI Sharing Technology
3 AI for Information Creation and Sharing
Information Creation & AI Sharing Technology
Automated Natural Lang. news writing Generation Sharing Content Machine Globally Translation Filtering Classification/ Misinformation Graph Neural Nets/ GANs 4 Why is NLG important?
Machine Writing Question Answering
ChatBOT Machine Translation
5 Machine Translation
6 AI to Improve Writing
Text generation to rescue! Gmail smart compose, smart reply.
7 8 Automated News Writing
Xiaomingbot is deployed and constantly producing news on social media platforms (Toutiao & TopBuzz).
9 human written
GPT3, edited by human
10 A New Working Style for Authors Human-AI Co-authoring
11 Outline
1. Motivation and Basics 2. Deep Latent Variable Models 3. Multimodal machine writing: show case 4. Summary
12 Modeling a Sequence
The quick brown fox jumps over the lazy dog . x = ( x1 , x2 , x3 , x4, x5 , x6 , x7, x8, x9, x10)
The central problem of language modeling is to find the joint probability distribution:
pθ(x) = pθ(x1, ⋯, xL)
There are many ways to represent and learn the joint probability model. 13 DGM Taxonomy
pθ(x) ⟷ pdata(x)
Maximum Likelihood Estimation Adversarial Learning GAN Explicit Density Implicit Density GSN Tractable Density Intractable Density Energy-based
Auto- Markov Parallel Latent Conditional Constrained Regressive Factorization Factorization Variable Model EBM PM Factorization RNN, LSTM Markov GLAT VAE CGMH Transformer Transformer NAT VTM MHA TSMH Transformer
Output
Softmax Multi-Head Scaled Dot-Product Attention Linear Linear MatMul Add & Norm Concat Feed Forward SoftMax
Add & Norm Scaled Dot-Product Mask (opt.) Add & Norm h Multi-Head Attention Attention Feed Forward Scale
Add & Norm LinearLinear LinearLinear LinearLinear Add & Norm MatMul Masked Multi-Head Multi-Head Q K V Attention Attention Q K V
Input Output Embedding Embedding
Inputs Outputs
Figure 1: The Transformer - model architecture.Figure 1: The Transformer - model architecture. 15
Decoder: The decoder is also composedDecoder: of a stackThe of decoderN =6identical is also composed layers. In of addition a stack to of theN =6 two identical layers. In addition to the two sub-layers in each encoder layer, the decodersub-layers inserts in each a third encoder sub-layer, layer, whichthe decoder performs inserts multi-head a third sub-layer, which performs multi-head attention over the output of the encoderattention stack. Similar over the to outputthe encoder, of the we encoder employ stack. residual Similar connections to the encoder, we employ residual connections around each of the sub-layers, followedaround by layer each normalization. of the sub-layers, We followed also modify by layer the self-attention normalization. We also modify the self-attention sub-layer in the decoder stack to preventsub-layer positions in the from decoder attending stack to to subsequent prevent positions positions. from This attending to subsequent positions. This masking, combined with fact that the outputmasking, embeddings combined are with offset fact bythat one the position, output embeddings ensures that are the offset by one position, ensures that the predictions for position i can depend onlypredictions on the known for position outputsi can at positions depend only less on than thei. known outputs at positions less than i.
3.2 Attention 3.2 Attention
An attention function can be describedAn as attention mapping function a query and can a be set described of key-value as mapping pairs to a an query output, and a set of key-value pairs to an output, where the query, keys, values, and outputwhere are the all query,vectors. keys, The values, output andis computed output are as all a weighted vectors. The sum output is computed as a weighted sum of the values, where the weight assignedof tothe each values, value where is computed the weight by assigned a compatibility to each function value is ofcomputed the by a compatibility function of the query with the corresponding key. query with the corresponding key.
3.2.1 Scaled Dot-Product Attention3.2.1 Scaled Dot-Product Attention
We call our particular attention "ScaledWe Dot-Product call our particular Attention" attention (Figure "Scaled 2). The Dot-Product input consists Attention" of (Figure 2). The input consists of queries and keys of dimension dk, andqueries values and of dimension keys of dimensiondv. We computedk, and thevalues dot of products dimension of thedv. We compute the dot products of the query with all keys, divide each by pdqueryk, and with apply all a keys, softmax divide function each by top obtaindk, and the apply weights a softmax on the function to obtain the weights on the values. values.
3 3 Deep Latent Variable Models for Text
• Disentangled Representation Learning for Text Generation [ICLR 20b, ACL 19c] • Interpretable Deep Latent Representation from Raw Text [ICML 20] • Mirror Generative Model for Neural Machine Translation [ICLR 20a]
16 Natural Language Descriptions
name Sukiyaki eatType pub Sukiyaki is a Japanese food Japanese restaurant. It is a pub and it has a price average average cost and rating good good rating. It is area seattle based in seattle.
17 Data to Text Generation
Data Table Sentence
area seattle 19 Our Motivation for Variational Template Machine Motivation 1: Continuous and Template disentangled representation for Sentence template and content Content Table
Motivation 2: Incorporate raw text corpus to learn good q (template, Raw text content | representation. sentence) 20 VTM [R. Ye, W. Shi, H. Zhou, Z. Wei, Lei Li, ICLR20b] Variational Template Machine
Input: triples of
• Semi-supervised learning: “Back-translate” corpus to obtain pseudo-parallel pairs
Ideal WIKI Ideal SPNLG 0.3 0.5 VTM T2S-beam T2S-beam 0.256 0.41 VTM T2S- T2S- 0.212 pretrain 0.32 pretrain
BLEU 0.23 BLEU 0.168
0.14 0.124 Temp-KN 0.05 Temp-KN 0.08 0.3 0.48 0.66 0.84 1.02 0.6 0.705 0.81 0.915 1.02 Self-BLEU Self-BLEU VTM uses beam-search decoding.
23 VTM [Ye, …, Lei Li, ICLR20b] VTM Generates Diverse Text
Input Data Table Generated Text
1: John Ryder (8 August 1889 – 4 April 1977) was an Australian cricketer. 2: Jack Ryder (born August 9, 1889 in Victoria, Australia) was an Australian cricketer. 3: John Ryder, also known as the king of Collingwood (8 August 1889 – 4 April 1977) was an Australian cricketer.
24 Learning Disentangled Representation of Syntax and Semantics DSSVAE enables learning and syntactic semantic transferring sentence-writing styles style content Syntax provider Semantic content
zsyn zsem There is an apple The dog is on the table behind the door
DSSVAE x
sentence There is a dog behind the door
25 DSS-VAE [Y. Bao, H. Zhou, S. Huang, Lei Li, L. Mou, O. Vechtomova, X. Dai, J. Chen, ACL19c] Interpretable Text Generation
Latent structure dialog actions GENERATOR Sampling “Remind me about x1 x2 x3 the football game.” [action=remind] “Will it be overcast tomorrow?” [action=request] x0 x1 x2 …… Generate Sentences with interpretable factors 26 How to Interpret Latent Variables in VAEs? Variational Auto-encoder (VAE) interpretable structure z x
(Kingma & Welling, 2013)
z: difficult to continuou interpret s latent variables discrete factors Will it be humid in New York today? Remind me about my meeting.
27 Discrete Variables Could Enhance Interpretability - but one has to do it right!
Gaussian Mixture Variational Auto- encoder (GM-VAE) interpretable structure c z x
(Dilokthanakul et al., 2016; Jiang et al., 2017)
: discrete c Why? mode- component How to fix it? collapse
z: continuous Will it be overcast Remind me latent variable tomorrow? about the
football game. 28 Do it right for VAE w/ hierarchical priors - Dispersed Exponential-family Mixture VAE The negative dispersion term in ELBO encourages the parameters of all mixture components in-distinguishable and induces the mode-collapse.
Dispersed EM-VAE
L(θ; x) = ELBO + β ⋅ Ld, Include an extra positive dispersion term to Ld = �q (c|x)A(ηc) − A(�q (c|x)ηc) . ϕ ϕ balance the mode collapse from ELBO 29 DEM-VAE [W. Shi, H. Zhou, N. Miao, Lei Li, ICML 2020] Generation Quality and Interpretability
DGM-VAE obtains the best performance in interpretability and reconstruction Homogeneity with golden label in DD BLEU of reconstruction in DD DI-VAE semi-VAE semi-VAE + Lmi GM-VAE GM-VAE + Lmi DGM-VAE DI-VAE semi-VAE DGM-VAE + Lmi semi-VAE + Lmi GM-VAE 0.24 GM-VAE + Lmi DGM-VAE DGM-VAE + Lmi 8 0.18 6 0.12 action 4 0.06 2 0 action emotion 0 BLEU Best interpretability Best reconstruction 30 DEM-VAE [W. Shi, H. Zhou, N. Miao, Lei Li, ICML 2020] Generate Sensible Dialog Response with DEM-VAE
Input Context Sys: “Taking you to Chevron.” sampling different values of discrete latent variables
(action = thanks) (action = request-address)
Predict Predict User: “Thank you car, let's go there!” User: “What is the address?”
Responses with different actions are generated by sampling different values of discrete latent variables.
31 DEM-VAE [W. Shi, H. Zhou, N. Miao, Lei Li, ICML 2020] Integrating Four Language Skills with MGNMT MGNM target src2t [p(y|z; θy)] [p(y|x, z; θxy)] x y z(x, y) y x source tgt2sr [p(x|z; θx)] [p(x|y, z; θyx)]
1. composing sentence in Source lang Benefits utilizing both 2. composing sentence in Target lang parallel bilingual data 3. translating from source to target and non- 4. translating from target to source parallel corpus 32 MGNMT [Z. Zheng, H. Zhou, S. Huang, L. Li, X. Dai, J. Chen, ICLR 2020a] y
src2tgt TM src LM [p(y|x, z)] [p(x|z)] x MGNMT y z(x, y) tgt LM tgt2src TM [p(y|z)] [p(x|y, z)]
x
Published as a conference paper at ICLR 2020 y
z z src2tgt TM src LM MGNMT [p(y|x, z)] [p(x|z)] target LM src2tgt TM [p(y|z; y)] [p(y|x, z; xy)] x y x y x y x MGNMT y z(x, y) z(x, y) GNMT MGNMT y x tgt LM tgt2src TM source LM tgt2src TM [p(x|z; )] [p(x|y, z; )] Figure 1: The [p(y|z)] [p(x|y, z)] x yx graphical model of MGNMT. Figure 2: Illustration of the mirror property of MGNMT. x For decoding, some related works (Gulcehre et al., 2015) propose to interpolate external language Mirror-Generativemodels LMy (trained separately NMT on target monolingual data) to translation model TMx y, which in- cludes knowledge from target monolingual data for better translation. This is particularly! useful for We proposedomain the adaptation mirror-generative because we may NMT obtain (MGnmt better translation) to address output thequite aforementioned fitting test domain problems (e.g., social networks), through a better LMy. However, directly interpolating an independent language for e↵ectivelymodel in exploiting decoding maybe non-parallel not the best. data First, in NMT. the language model used here is external, still inde- pendently learned to the translation model, thus the two models may not cooperate well by a simple I Jointly training translation models (i.e., Tmx y and Tmy x) and language models (i.e., interpolation mechanism (even conflict). Additionally,! the language! model is only included in de- Approach:Lmcoding,x and whichLmy is) not in a considered unified probabilistic inMirror-Generative training. This framework. leads to the inconsistency of training and NMT decoding, I Alatentvariablewhich may harm thez performance.representing the shared semantic space between the two language x andIn thisy, inspired paper, we by propose generative the mirror-generative NMT (Shaw NMT et al., (MG 2018)NMT) to address the aforementioned problems• The for effectively mirror exploiting property non-parallel to data decompose in NMT. MGNMT is proposed to jointly train translation models (i.e., TMx y and TMy x) and language models (i.e., LMx and LMy) in a unified framework, which is non-trivial.! Inspired! by generativeMGNMT NMT (Shah & Barber, 2018), we propose to introduce a latent semantic variable ztargetshared LM betweensrc2tgtx and TyM. Our method exploits the symmetry,
or mirror property, in decomposing the[p(y conditional|z; y)] joint[p(y| probabilityx, z; xy)] p(x, y z), namely: x y| log p(x, y z) = log p(x z) + log pz((yx,x,y) z) = log p(y z) + log p(x y, z) | y | | | x | 1 = [log p(sourcey x, z L)M+ log p(ytgt2srcz) + log TM p(x y, z) + log p(x z)] (1) 2 [p(|x|z; x)] [p|(x|y, z; yx)] | | src2tgt TMx y target LMy tgt2src TMy x source LMx ! ! The graphical model of MGNMT| is illustrated{z } | in Figure{z } 1.| MGNMT{z aligns} | the{z bidirectional} trans- Zhenglation et al. models (NJU & ByteDance) as well as language models in twoiclr20 languages through a shared latent semantic31 Mar, space 2020 13 / 34 (Figure 2), so that all ofp( themx, y| arez) = relevantp(y|x and, z)p become(x|z) = conditionalp(x|y, z independent)p(x|z) given z. In such case, MGNMT enables following advantages: • Relevant TMs & LMs under a unified probabilistic (i) For training, thanks to z as a bridge, TMy x and TMx y are not independent, thus ev- framework!ery updating of one direction will directly! benefit the other! direction. This improves the efficiency of using non-parallel data. (Section 3.1) (ii)–ForEnables decoding, MG theNMT aforementionedcould naturally take advantages advantages of its internal target-side language model, which is jointly learned with the translation model. Both of them contribute to33 the better generation process together. (Section 3.2) Note that MGNMT is orthogonal to dual learning (He et al., 2016a) and joint back-translation (Zhang et al., 2018). Translation models in MGNMT are dependent, and the two translation models could directly promote each other. Differently, dual learning and joint back-translation works in an im- plicit way, and these two approaches can also be used to further improve MGNMT. The language models used in dual learning faces the same problem as Gulcehre et al. (2015). Given GNMT, the proposed MGNMT is also non-trivial. GNMT only has a source-side language model, thus it cannot enhance decoding like MGNMT. Also, in Shah & Barber (2018), they require GNMT to share all the parameters and vocabularies between translation models so as to utilize monolingual data, which is not best suited for distant language pairs. We will give more comparison in the related work. Experiments show that MGNMT achieves competitive performance on parallel bilingual data, while it does advance training on non-parallel data. MGNMT outperforms several strong baselines in different scenarios and language pairs, including resource-rich scenarios, as well as resource-poor circumstances on low-resource language translation and cross-domain translation. Moreover, we show that translation quality indeed becomes better when the jointly learned translation model and language model of MGNMT work together. We also demonstrate that MGNMT is architecture-free which can be applied to any neural sequence model such as Transformer and RNN. These pieces of evidence verify that MGNMT meets our expectation of fully utilizing non-parallel data.
2 Published as a conference paper at ICLR 2020
Table 2: Statistics of the training datasets for each translation tasks. These values of D [q(z) p(z)] KL || are to someMGNMT extent large, which makes means that MG betterNMT does rely onuse the latent of variable. non- Dataset WMT14 EN DE NIST EN ZH WMT16 EN RO IWSLT16 EN DE $ $ $ $ KL-annealing steps 35kparallel 13.5k data8k 4k D [q(z) p(z)] 6.78 8.26 6.36 7.81 KL ||
Table• 3:Low BLEU scores resource on low-resource results translation (WMT16 EN RO), and cross-domain translation (IWSLT EN DE). $ $
LOW-RESOURCE CROSS-DOMAIN Model WMT16 EN RO IN-DOMAIN (TED)OUT-DOMAIN (NEWS) $ EN-RO RO-EN EN-DE DE-EN EN-DE DE-EN Transformer (Vaswani et al., 2017) 32.1 33.2 27.5 32.8 17.1 19.9 GNMT (Shah & Barber, 2018) 32.4 33.6 28.0 33.2 17.4 20.1 GNMT-M-SSL + non-parallel (Shah & Barber, 2018) 34.1 35.3 28.4 33.7 22.0 24.9 Transformer+BT + non-parallel (Sennrich et al., 2016b) 33.9 35.0 27.8 33.3 20.9 24.3 Transformer+JBT + non-parallel (Zhang et al., 2018) 34.5 35.7 28.4 33.8 21.9 25.1 Transformer+Dual + non-parallel (He et al., 2016a) 34.6 35.7 28.5 34.0 21.8 25.3 MGNMT 32.7 33.9 28.2 33.6 17.6 20.2 MGNMT + non-parallel 34.9 36.1 28.5 34.2 22.8 26.1
4.1 RESULTS AND DISCUSSION
As shown in Table 3 and Table 4, MGNMT outperforms our competitive Transformer base- line (Vaswani et al., 2017), Transformer-based GNMT (Shah & Barber, 2018) and related work in both resource-poor scenarios and resource-rich scenarios. 34
MGNMT makes better use of non-parallel data. As shown in Table 3, MGNMT outperforms our competitive Transformer baseline (Vaswani et al., 2017), Transformer-based GNMT (Shah & Barber, 2018) and related work in both resource-poor scenarios. 1. On low-resource language pairs. The proposed MGNMT obtains a bit improvement over Trans- former and GNMT on the scarce bilingual data. Large margins of improvement are obtained by exploiting non-parallel data. 2. On cross-domain translation. To evaluate the capability of our model in the cross-domain set- ting, we first trained our model on TED data from IWSLT benchmark as in-domain training, and then exposed the model to out-of-domain NEWS non-parallel bilingual data from News Crawl to access- ing out-domain knowledge. As shown in Table 3, being invisible to out-domain training data leads to poor performance in out-domain testset of both Transformer and MGNMT. In this case, out-domain non-parallel data contributes significantly, leading to 5.7 6.4 BLEU gains. We also conduct a case study on the cross-domain translation in Appendix. ⇠ 3. On Resource-rich scenarios. We also conduct regular translation experiments on two resource- rich language pairs, i.e., EN DE and NIST EN ZH. As shown in Table 4, MGNMT can obtain comparable results compared$ to discriminative baseline$ RNMT and generative baseline GNMT on pure parallel setting. Our model can also achieve better performance by the aid of non-parallel bilingual data than the compared previous approaches, consistent with the experimental results in resource-poor scenarios. 4. Comparison to other semi-supervised work. We compare our approach with well-established approaches which are also designed for leveraging non-parallel data, including back-translation (Sennrich et al., 2016b, Transformer+BT), joint back-translation training (Zhang et al., 2018, Trans- former+JBT), multi-lingual and semi-supervised variant of GNMT (Shah & Barber, 2018, GNMT- M-SSL), and dual learning (He et al., 2016a, Transformer+Dual). As shown in Table 3, while introducing non-parallel data to either low-resource language or cross-domain translation, all listed semi-supervised approaches gain substantial improvements. Among them, our MGNMT achieves the best BLEU score. Meanwhile, in resource-rich language pairs, the results are consistent. We suggest that because the jointly trained language model and translation model could work coordi- nately for decoding, MGNMT surpasses joint back-translation and dual learning. Interestingly, we can see that the GNMT-M-SLL performs poorly on NIST EN ZH, which means parameters-sharing is not quite suitable for distant language pair. These results indicate$ its promising strength of boost-
7 mRASP: Multilingual Machine Translation
Pre-training J’adore chanter et danser
Encoder Decoder Orig tok
pos 0 1 2 3 4 5 0 1 2 3 4
Random Aligned Substitution
De It 35 mRASP gets universal improvement
Extremely-Low Resource Direc ons Direct mRASP 40 35.8 32.3 28.6 31.1 30.4 30 25.8 25.3 27 20 8.5 9.6 10.2 8.3 10 5.4 7.2 4.9 6.7 0 En2Be Be2En En2My My2En En2Af Af2En En2Eo Eo2En Low Resource Direc ons 50 44.6 39 37.4 40 32.4 33.3 27.6 30.5 29.2 29.8 30 23.2 21 19.4 22.7 20 19 19 10.7 10 0 En2He He2En En2Tr Tr2En En2Ro Ro2En En2Cs Cs2En 36 lower resource higher resource mRASP gets universal improvement
• Rich resource benchmarks can be further improved (En->Fr +1.1BLEU). 31 47 Direct CTNMT Direct CTNMT XLM MASS mBART mRASP mBERT mRASP 30.25 45.25
29.5 43.5
28.75 41.75
28 40 37 En2De(wmt2016) En2Fr(wmt2014) VolcTrans fanyi.volcengine.cn
50+ 9 Billion 16 Clients languages
Public MT Corpus 3rd-party best VolcTrans 60
45
30 BLEU 15
0 38 fr->enen->fr en->itit->en de->enen->dees->enen->esko->enen->koen->zhzh->enja->enen->jazh->jaja->zhen->thvi->enen->viid->enen->idar->enen->aren->ruru->en en->hihi->en Speech-to-Text Translation
VolcTrans
39 Simultaneous Speech-to-text Translation @ VolcTrans Multimodal Machine Writing
Xiaomingbot [R. Xu, J. Cao, M. Wang, J. Chen, H. Zhou, Y. Zeng, Y. Wang, L. Chen, X. Yin, X. Zhang, S. Jiang, Y. Wang, Lei Li, ACL 2020] GraspSnooker [Z. Sun, J. Chen, H. Zhou, D. Zhou, Lei Li, M. Jiang, IJCAI19b] Jersey Number Recognition with Semi-Supervised Spatial Transformer Network [G. Li, S. Xu, X. Liu, Lei Li, C. Wang, CVPR-CVS18] Automatic News Writing in Real-world
• Tencent: Dreamwriter, started in 2015.9 • Fast Writer Xiaoxin: Xinhuanet, started in 2015.11 • Xiaomingbot: ByteDance, started in 2016.8 • Xiaonan: Southern Weekend, started 2017.1 • Wibbitz: USA Today • Heliograf: Washington Post
41 Xiaomingbot Automatic News Writing System Winning 2017 Wu Wen-tsün Award in AI from CAAI 600,000 articles
6 lang
150,000 followers
42 Xiaomingbot : Multilingual Robot News Reporter
43 Runxin Xu, Jun Cao, Mingxuan Wang, Jiaze Chen, Hao Zhou, Ying Zeng, Yuping Wang, Li Chen, Xiang Yin, Xijin Zhang, Songcheng Jiang, Yuxuan Wang, Lei Li, ACL 2020. Snooker Commentary Generation Combining Visual Understanding with Strategy Prediction
44 GraspSnooker [Z. Sun, J. Chen, H. Zhou, D. Zhou, Lei Li, M. Jiang, IJCAI19b] Summary
• Transformer, LSTM & Softmax: Basic neural generation nets for text • Disentangled Latent Representation – VTM: Learning Latent Templates in Variational Space – DSS-VAE: Disentangled syntax and semantic representation • DEM-VAE: Self identifying meaningful clusters with corpus • MGNMT: – integrate four language capabilities together – Utilize both parallel and non-parallel corpus • Multimodal Machine Writing – Xiaomingbot system: 600k articles and 150k followers • Deployed in multiple online platforms and used by over 100 millions of users 45 Recap: DGM Taxonomy
pθ(x) ⟷ pdata(x)
Maximum Likelihood Estimation Adversarial Learning GAN Explicit Density Implicit Density GSN Tractable Density Intractable Density Energy-based
Auto- Markov Parallel Latent Conditional Constrained Regressive Factorization Factorization Variable Model EBM PM Factorization RNN, LSTM Markov Glancing VAE CGMH Transformer Transformer Transformer VTM MHA NAT TSMH Thanks
• Joint w/ Hao Zhou, Rong Ye, Ning Miao, Wenxian Shi, Zaixiang Zheng, Huangzhao Zhang, Ying Zeng, Jiaze Chen, Han Zhang • Contact: lileilab@bytedance.com
Multilingual MT Pretraining https://github.com/linzehui/mRASP
A high performance sequence processing lib 47 https://github.com/bytedance/lightseq Reference
1. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin. Attention Is All You Need. NeurIPS 2017. 2. Ning Miao, Hao Zhou, Lili Mou, Rui Yan, Lei Li. “CGMH: Constrained Sentence Generation by Metropolis-Hastings Sampling”. In: the 33rd AAAI Conference on Artificial Intelligence (AAAI). Jan. 2019. 3. Huangzhao Zhang, Ning Miao, Hao Zhou, Lei Li. “Generating Fluent Adversarial Examples for Natural Languages”. In: the 57th Annual Meeting of the Association for Computational Linguistics (ACL) - short papers. July 2019. 4. Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xinyu Dai, Jiajun Chen. “Generating Sentences from Disentangled Syntactic and Semantic Spaces”. In: the 57th Annual Meet- ing of the Association for Computational Linguistics (ACL). July 2019. 5. Ning Miao, Hao Zhou, Chengqi Zhao, Wenxian Shi, Lei Li. “Kernelized Bayesian Softmax for Text Generation”. In: the 33rd Conference on Neural Information Processing Systems (NeurIPS). Dec. 2019. 6. Zaixiang Zheng, Hao Zhou, Shujian Huang, Lei Li, Xinyu Dai, Jiajun Chen. “Mirror Generative Models for Neural Machine Translation”. In: International Conference on Learning Representations (ICLR). Apr. 2020. 7. Rong Ye, Wenxian Shi, Hao Zhou, Zhongyu Wei, Lei Li. “Variational Template Machine for Data- to-Text Generation”. In: International Conference on Learning Representations (ICLR). Apr. 2020. 8. Ning Miao, Yuxuan Song, Hao Zhou, Lei Li. “Do you have the right scissors? Tailoring Pre-trained Language Models via Monte-Carlo Methods”. In: the 58th Annual Meeting of the Association for Computational Linguistics (ACL) - short papers. July 2020. 9. Wenxian Shi, Hao Zhou, Ning Miao, Lei Li. “Dispersing Exponential Family Mixture VAEs for Interpretable Text Generation”. In: Proceedings of the 37th International Conference on Machine learning (ICML). July 2020. 10.Maosen Zhang, Nan Jiang, Lei Li, Yexiang Xue. “Constraint Satisfaction Driven Natural Language Generation: A Tree Search Embedded MCMC Approach”. In: the Conference on Empirical Methods in Natural Language Processing (EMNLP) - Findings. Nov. 2020. 11. Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng, Hao Zhou, Lei Li. Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information. EMNLP 2020.
48