Topological for Sentence Ordering

Shrimai Prabhumoye, Ruslan Salakhutdinov, Alan W Black Sentence Ordering Task Sentence Ordering Task Before

Ironman steals the Infinity Stones back from Thanos and uses them to disintegrate Thanos and his army, the cost of his life.

Hulk travels to New York City in 2012 and convinces the Ancient One to give him the Stone.

Thor decapitates Thanos.

Five years later, AntMan escapes from the quantum realm.

Ironman builds a time machine to save the world. Sentence Ordering Task Before After

Ironman steals the Infinity Stones back Thor decapitates Thanos. from Thanos and uses them to disintegrate Thanos and his army, at the cost of his life. Five years later, AntMan escapes from the quantum realm. Hulk travels to New York City in 2012 and convinces the Ancient One to give him the Ironman builds a time machine to save the Time Stone. world.

Thor decapitates Thanos. Hulk travels to New York City in 2012 and convinces the Ancient One to give him the Five years later, AntMan escapes from Time Stone. the quantum realm. Ironman steals the Infinity Stones back Ironman builds a time machine to save the from Thanos and uses them to disintegrate world. Thanos and his army, at the cost of his life. Methodology

● Constraint Solving Problem vs Sequence Prediction Task For a document with sentences ● n ({s1…sn}) n ● |!| = constraints (2) Predicted constraints of the form ● s1 < s2 ● 4 sentences in a document then 6 constraints

● {s1 < s2, s1 < s3, s1 < s4, s2 < s3, s2 < s4, s3 < s4} ● Topological sort to find an order given ! Graph: if ● s1 → s2 s1 < s2 Constraint Learning

● BERT based Representation (B-TSort) ● Next Sentence Prediction MLP(BERT( )) ● s1[#$%]s2

● LSTM based Representation (L-TSort) ; ● h1 = LSTM(s1) h2 = LSTM(s2) MLP( ) ● [h1; h2] Baselines

● Attention Order Network (AON) ● LSTM: sentence representation ● Transformer: document representation ● LSTM decoder: generate order

● BERT Attention Order Network (B-AON) ● BERT: sentence representation Results for NIPS abstracts

AON L-TSort B-AON B-TSort

90 87.97 83.65 83.45 81 80.97 80.08 76.29 73 74.38 71.11 67 64 67.5 61.48

55.23 50.5 45 43.08

32.59

22.5 19.9 16.25 12.19

0 Perfect Match Sentence Acc Kendall Tau Rouge-S LCS Results for NIPS abstracts

AON L-TSort B-AON B-TSort

90 87.97 83.65 83.45 81 80.97 80.08 76.29 73 74.38 71.11 67 64 67.5 61.48

entire 55.23 sequence was 50.5

correctly 43.08 45 predicted 32.59

22.5 19.9 16.25 12.19

0 Perfect Match Sentence Acc Kendall Tau Rouge-S LCS Results for NIPS abstracts

AON L-TSort B-AON B-TSort

90 87.97 83.65 83.45 81 80.97 80.08 76.29 73 74.38 absolute position was 71.11 67 correctly predicted 64 67.5 61.48

55.23 50.5 45 43.08

32.59

22.5 19.9 16.25 12.19

0 Perfect Match Sentence Acc Kendall Tau Rouge-S LCS Results for NIPS abstracts

AON L-TSort B-AON B-TSort number of inversions to 90 87.97 reach correct order 83.65 83.45 81 80.97 80.08 76.29 73 74.38 71.11 67 64 67.5 61.48

55.23 50.5 45 43.08

32.59

22.5 19.9 16.25 12.19

0 Perfect Match Sentence Acc Kendall Tau Rouge-S LCS Results for NIPS abstracts

AON L-TSort B-AON B-TSort pairs with correct relative order 90 87.97 83.65 83.45 81 80.97 80.08 76.29 73 74.38 71.11 67 64 67.5 61.48

55.23 50.5 45 43.08

32.59

22.5 19.9 16.25 12.19

0 Perfect Match Sentence Acc Kendall Tau Rouge-S LCS Results for NIPS abstracts

AON L-TSort B-AON B-TSort longest common 90 87.97 subsequence 83.65 83.45 81 80.97 80.08 76.29 73 74.38 71.11 67 64 67.5 61.48

55.23 50.5 45 43.08

32.59

22.5 19.9 16.25 12.19

0 Perfect Match Sentence Acc Kendall Tau Rouge-S LCS Results for NIPS abstracts

AON L-TSort B-AON B-TSort

90 • L-TSort performs close to AON 87.97 83.65 83.45 81 80.97 80.08 76.29 73 74.38 71.11 67 64 67.5 61.48

55.23 50.5 45 43.08

32.59

22.5 19.9 16.25 12.19

0 Perfect Match Sentence Acc Kendall Tau Rouge-S LCS Results for NIPS abstracts

AON L-TSort B-AON B-TSort

90 87.97 83.65 83.45 • B-AON performs better than AON 81 80.97 80.08 76.29 73 74.38 71.11 67 64 67.5 61.48

55.23 50.5 45 43.08

32.59

22.5 19.9 16.25 12.19

0 Perfect Match Sentence Acc Kendall Tau Rouge-S LCS Results for NIPS abstracts

AON L-TSort B-AON B-TSort

90 87.97 83.65 83.45 81 80.97 80.08 76.29 73 74.38 • B-TSort performs the best 71.11 67 64 67.5 61.48

55.23 50.5 45 43.08

32.59

22.5 19.9 16.25 12.19

0 Perfect Match Sentence Acc Kendall Tau Rouge-S LCS Results for NIPS abstracts

AON L-TSort B-AON B-TSort

90 87.97 83.65 83.45 81 80.97 80.08 76.29 73 74.38 71.11 67 64 67.5 61.48

55.23 50.5 45 43.08

32.59

22.5 19.9 16.25 12.19

0 Perfect Match Sentence Acc Kendall Tau Rouge-S LCS Results for Human Evaluation

B-TSort No Pereference B-AON Gold

B-TSort vs B-AON 41 28 31

B-TSort vs Gold 26 20 54

B-AON vs Gold 22 24 54

0 25 50 75 100 Results

● B-TSort performs best in all metrics for SIND captions, NSF

abstract, AAN abstract datasets

● Analysis of displaced sentences ● distance between the predicted position and the actual position of a sentence. ● B-TSort displaces less sentences. ● Analysis of documents with than 10 sentences ● B-TSort also performs better for longer sentences ● Percentage of mismatch in input and output for AON ● Order generation can have a mismatch between the input sentences and the generated orders. ● This problem does not arise for our models.