Topic Segmentation: Algorithms and Applications

University of Pennsylvania ScholarlyCommons

IRCS Technical Reports Series Institute for Research in Cognitive Science

8-1-1998

Topic Segmentation: Algorithms And Applications

Jeffrey C. Reynar University of Pennsylvania

Follow this and additional works at: https://repository.upenn.edu/ircs_reports

Part of the Databases and Information Systems Commons

Reynar, Jeffrey C., "Topic Segmentation: Algorithms And Applications" (1998). IRCS Technical Reports Series. 66. https://repository.upenn.edu/ircs_reports/66

University of Pennsylvania Institute for Research in Cognitive Science Technical Report No. IRCS-98-21.

This paper is posted at ScholarlyCommons. https://repository.upenn.edu/ircs_reports/66 For more information, please contact [email protected]. Topic Segmentation: Algorithms And Applications

Abstract Most documents are aboutmore than one subject, but the majority of natural language processing algorithms and information retrieval techniques implicitly assume that every document has just one topic. The work described herein is about clues which mark shifts to new topics, algorithms for identifying topic boundaries and the uses of such boundaries once identified.

A number of topic shift indicators have been proposed in the literature. We review these features, suggest several new ones and test most of them in implemented topic segmentation algorithms. Hints about topic boundaries include repetitions of character sequences, patterns of word and word n-gram repetition, word frequency, the presence of cue words and phrases and the use of synonyms.

The algorithms we present use cues singly or in combination to identify topic shifts in several kinds of documents. One algorithm tracks compression performance, which is an indicator of topic shift because self-similarity within topic segments should be greater than between-segment similarity. Another technique relies on word repetition and places boundaries by minimizing word repetitions across segment boundaries. A third method compares the performance of a language model with and without knowledge of the contents of preceding sentences to determine whether a topic shift has occurred. We use the output of this algorithm in a statistical model which incorporates synonymy, bigram repetition and other features for topic segmentation.

We benchmark our algorithms and compare them to algorithms from the literature using concatenations of documents, and then perform further evaluation of our techniques using a collection of news broadcasts transcribed both by annotators and using a speech recognition system. We also test the effectiveness of our algorithms for identifying both chapter boundaries in works of literature and story boundaries in Spanish news broadcasts.

We suggest ways to improve information retrieval, language modeling and various natural language processing algorithms by exploiting the topic segmentation.

Disciplines Databases and Information Systems

Comments University of Pennsylvania Institute for Research in Cognitive Science Technical Report No. IRCS-98-21.

This thesis or dissertation is available at ScholarlyCommons: https://repository.upenn.edu/ircs_reports/66

TOPIC SEGMENTATION ALGORITHMS AND APPLICATIONS

Jerey C Reynar

A DISSERTATION

Computer and Information Science

Presented to the Faculties of the UniversityofPennsylvania

in Partial Fulllment of the Requirements for the Degree of Do ctor of Philosophy

Mitchell P Marcus

Adviser

Mark Steedman Graduate Group Chair

Jerey C Reynar

In memory of Mom and Dad iii

Acknowledgements

Like all research the work describ ed in this dissertation was not conducted in isolation

It is imp ossible to thank everyone who has in some way contributed to it and equally

imp ossible to thank individuals for all of their contributions I am indebted for technical

insights encouragement and friendship to manymemb ers of the vibrant research commu

nity at the UniversityofPennsylvania as well as those I had the privilege of working with

at the Summer Institute of Linguistics IBMs TJ Watson Research Center Fuji Xerox

and Infonautics

I rst want to thank my advisor Mitch Marcus for providing me with many good

ideas inspiring others in me and helping me rene and develop my own Thanks also to

Mitchforallowing me the freedom to pursue the MUC and LexisNexis pro jects and to

sp end several summers working in industry

Thanks to the memb ers of my thesis committee Julia Hirschb erg Aravind Joshi Mark

Lib erman and Lyle Ungar for providing suggestions which improved the quality of this

work Additional thanks to Aravind Mark and Mitch for serving on my WPE commit

tee

Ive b eneted from conversations with and used natural language pro cessing to ols de

velop ed by the memb ers of the UniversityofPennsylvania MUC team Breck Baldwin

Mike Collins Jason Eisner Adwait Ratnaparkhi Joseph Rosenzweig Ano op Sarkar and

B Srinivas Adwait deserves sp ecial thanks for p ermitting me to use his maximum entropy

Chandrasekar Christy Doran Beth Ann Ho ckey mo deling to ols Thanks also to Mickey

Al Kim Dan Melamed Tom Morton Michael Niv Martha Palmer MikeSchultz iv

Matthew Stone and David Yarowsky for helpful discussions and feedback My time at

Penn was made more enjoyable by the friendship of each of these p eople as well

I also greatly proted from discussions with researchers outside of Penn including Ken

Church Lance Ramshaw Salim Roukos Penni Sibun Larry Spitz and Mark Wasson

Thanks also to Kathy McCoy my de facto undergraduate advisor for intro ducing me to

computational linguistics

Thanks to Betsy Norman and Amy Dunn for regularly squeezing me into Mitchs

schedule on short notice and for go o d conversation while waiting to see him Im grateful

to MikeFelker for help getting over the administrativehurdles I faced as a graduate student

and to the sta of the CIS business oce for helping me navigate the complexities of the

UniversityofP ennsylvanias payroll and billing systems

FinallyI owe the most gratitude to my parents who always encouraged me to pursue

my dreams and strive to do my b est and to my wife Angela for picking up where they

left o v

Abstract

TOPIC SEGMENTATION ALGORITHMS AND APPLICATIONS

Jerey C Reynar

Sup ervisor Mitchell P Marcus

Most do cuments are about more than one sub ject but the ma jority of natural language

pro cessing algorithms and information retrieval techniques implicitly assume that every

do cument has just one topic The work describ ed herein is ab out clues which mark shifts

to new topics algorithms for identifying topic b oundaries and the uses of such b oundaries

once identied

A number of topic shift indicators have been prop osed in the literature We review

these features suggest several new ones and test most of them in implemented topic

segmentation algorithms Hints ab out topic b oundaries include rep etitions of character

sequences patterns of word and word ngram rep etition word frequency the presence of

cue words and phrases and the use of synon yms

The algorithms we present use cues singly or in combination to identify topic shifts

in several kinds of do cuments One algorithm tracks compression p erformance which is

an indicator of topic shift b ecause selfsimilarity within topic segments should b e greater

than betweensegment similarity Another technique relies on word rep etition and places

b oundaries by minimizing word rep etitions across segment b oundaries A third metho d

compares the p erformance of a language mo del with and without knowledge of the contents

of preceding sentences to determine whether a topic shift has o ccurred We use the output vi

of this algorithm in a statistical mo del which incorp orates synonymy bigram rep etition

and other features for topic segmentation

We b enchmark our algorithms and compare them to algorithms from the literature

using concatenations of do cuments and then p erform further evaluation of our techniques

using a collection of news broadcasts transcrib ed b oth by annotators and using a sp eech

recognition system We also test the eectiveness of our algorithms for identifying both

chapter b oundaries in works of literature and story b oundaries in Spanish news broadcasts

We suggest ways to improve information retrieval language mo deling and various nat

ural language pro cessing algorithms by exploiting the topic segmentation vii

Contents

Acknowledgements iv

Abstract vi

Intro duction

Motivation

Segmentation Cues

The Noisy Channel Mo del

Goals

Topic Segmentation not Discourse Segmentation

Outline

Theories of Discourse Structure

Halliday and Hasan

Grosz Sidner

Skoro cho dko

Structuring Clues

Currently Uncomputable Features

Grimes

Nakhimovsky

Summary

Currently Computable Features

Cue Words Phrases viii

First Uses

Word Rep etition

Word ngram Rep etition

W ord Frequency

Synonymy

Named Entities

Pronoun Usage

Character ngram Rep etition

Additional Indicators

Previous Text Segmentation Metho ds

Word Rep etition

Morris Hirst

Hearst

Richmond Smith Amitay

Yaari

Other Approaches

Other Features

Beeferman Berger Laerty

Phillips

Youmans

Kozima

Ponte Croft

Discussion

Algorithms

Desiderata

Text Normalization

Tokenization

Conversion to Lowercase

Lemmatization ix

Identifying ClosedClass Words

Do cument Concatenations

Performance of Algorithms from the Literature

TextTiling

Vo cabulary Management Proles

Compression Algorithm

Lemp elZiv

Complicating Factors

Evaluation

Optimization Algorithm

Dotplotting

Dotplots for Text Segmentation

Algorithmic Boundary Identication

Minimization versus Maximization

Formal Description of the Algorithm

SimilaritytoVector Space Mo del

Evaluation

Word Frequency Algorithm

The G Mo del

Word Frequency Algorithm Sp ecication

The Impact of Individual Words

The Eect of Perturbing the Parameters

Parameter Estimation

Evaluation

No Training Corpus Required

A Maximum Entropy Mo del

Performance Reprise

Evaluation

Performance Measures

Comparison to human annotation x

Broadcast News

Topic Detection Tracking Corpus

Performance on the TDT Corpus

Adapting Algorithm WF for TDT Data

Inducing Domain Cues

The Value of Dierent Indicators

Recovering Authorial Structure

Conclusions

Applications

Information Retrieval

Previous Work

TREC Sp oken Do cument Retrieval Task

Language Mo deling

Text Segmentation and Language Mo deling

TopicDep endent Language Mo deling

Improving NLP Algorithms

Potential Applications

Summarization

Hyp ertext

Information Extraction

Topic Detection and Tracking

Automated Essay Grading

Conclusions

Future Directions

Bibliography xi

List of Tables

Cue words from Hirschb erg and Litman

Domain cues we identied by hand using do cuments from the HUB

broadcast news corpus

List of corp orate designators used for named entity recognition

The typ es of clues used byvarious text structuring algorithms

The features used by our text structuring algorithms

Closed class words b eginning with the letter a

Accuracy of several baseline segmentation algorithms on pairs of con

catenated WSJ articles The last two rows are average results for ve iter

ations of the random selection algorithm

TextTiling applied to do cument concatenations TextTiling did not lab el

any b oundaries for of the concatenations b ecause they w ere to o short

Results of our implementation of algorithms based on the vector space mo del

when applied to do cuments without optional normalization Each algorithm

was tested on concatenations of pairs of Wal l Street Journal articles

The row in gray presents results with the same settings as TextTiling

Results of our implementation of algorithms based on the vector space mo del

when applied to normalized data Each algorithm was tested on con

catenations of pairs of Wal l Street Journal articles The rowingray presents

results with the same settings as TextTiling xii

Results of several variants of Youmans technique when applied to data

consisting of concatenations of pairs of Wal l Street Journal articles

Sample LZ compression

Results of the compression algorithm on concatenations of pairs of Wal l

Street Journal articles

Sample word rep etition matrix

The application of two optimization algorithms for topic segmentation to

the sample text x xy x y

Results of manyvariants of our optimization algorithm when tested on

concatenations of pairs of Wal l Street Journal articles We did not reduce

words to their ro ots or ignore frequentwords

Results of variants of our optimization technique when tested on con

catenations of pairs of Wal l Street Journal articles The data was prepro

cessed to reduce words to their ro ots and ignore frequentwords

Example distribution of burstywords in a do cument

Conditional probabilities of seeing a particular number of o ccurrences of a

word in blo ck given that a certain number have been observed in blo ck

k is the number of o ccurrences in blo cks and combined M is a

normalization constant discussed in the text

one w

The eect of increasing parameter values on the ratio of

two w

Range of parameters for the G mo del estimated from the Wal l Street Journal

training corpus with Go o dTuring smo othing applied

Results of the word frequency algorithm when given guess ab out the lo

cation of a b oundary The data consisted of concatenations of pairs of

Wal l Street Journal articles

Results of the word frequency algorithm when only the parameters for un

of pairs known words were used The data consisted of concatenations

of Wal l Street Journal articles

Pronouns used as indicators of topic b oundaries

News programs found in the HUB Broadcast News Corpus xiii

Statistics ab out the HUB corpus annotation

The interannotator reliability of the annotation of the HUB corpus

Performance of various algorithms on les from the HUB Broad

cast News Corpus

Performance of algorithms WF and ME on the test p ortion of the TDT

corpus

The b est cue phrases induced from the TDT training corpus

Performance of algorithm WF mo died to select the b est b oundary among

neighb oring hyp othesized b oundaries and ME trained on cues induced from

the TDT training corpus

Performance of algorithm ME when trained on the training p ortion of the

TDT corpus using all indicators of text structure except the one listed in

each row

Accuracy of the algorithm WF on several works of literature Columns

lab eled Acc indicate accuracy while those lab eled Prob show p erformance

using the probabilistic metric of Beeferman et al

IR p erformance on the sp oken do cument retrieval corpus with stemming

using SMARTs built in stemmer

Results of a language mo deling exp eriment conducted on the TDT corpus

Number of candidate antecedents for singular pronouns that referred to

people in randomly selected broadcast news transcripts pronouns

were examined xiv

List of Figures

The noisy channel mo del

Changes in the attentional intentional and linguistic structure when pro

cessing a sample text

Skoro cho dkos four text typ es

Example of a domain cue marking the b oundary b etween topic segments in

afragment of a transcript of an episo de of National Public Radios show All

Things Considered The phrase in b old is a domainsp ecic cue phrase of

the form Im PERSON The gap separates two dierent news stories

Features used by our named entity recognizer when determining the lab el

for the word apple in Example

Example of a large number of rst uses of words marking a new segment

This is a fragment of a transcript of an episo de of National Public Radios

show Al l Things Considered Words in b old are used for the rst time The

gap is b etween two news items

Example of a large numb er of rst uses of op enclass words marking a new

segment This is a fragment of a transcript of an episo de of National Public

Radios show All Things Considered Words in bold are used for the rst

time The gap separates two news stories

Example showing the degree of word rep etition within topic segments The

data is from the National Public Radio program All Things Considered xv

Example demonstrating the quantityofcontentword rep etition within topic

segments The text is from the National Public Radio program All Things

Considered

Example indicating the usefulness of tracking the rep etition of word bigrams

for topic segmentation The data is from the National Public Radio program

Al l Things Considered

Example showing the usefulness of tracking the rep etition of content word

bigrams for topic segmentation The text is transcrib ed from the National

Public Radio program Al l Things Considered

Example indicating the utility of tracking synonyms for identifying topic

b oundaries The data is from the National Public Radio program Al l Things

Considered

Example transcript indicating the usefulness of coreference for topic segmen

tation The text is transcrib ed from the National Public Radio program All

Things Considered

Example showing the usefulness of limited coreference between named en

tities The text is a transcript of the National Public Radio program All

Things Considered

Example of a text region b eginning with a pronoun which contraindicates

the existence of a preceding segment b oundary This is a fragment of a tran

script of an episo de of National Public Radios show Al l Things Considered

The crucial pronoun is shown in b old There is no topic b oundary b efore

the sentence b eginning with he said nal ly

Example showing the utility of tracking character ngram rep etition for

Public identifying topic b oundaries This is a transcript of the National

Radio program Al l Things Considered

Unsmo othed depth score graph of two concatenated Wal l Street Journal

articles

Smo othed depth score graph of two concatenated Wal l Street Journal articles xvi

Sample dendogram of the typ e pro duced by Yaaris hierarchical clustering

algorithm The xaxis is the sentence numb er and the y axis is the level of

the merge

Typ etoken plot of Brown corpus le cd

Vo cabulary Management Prole of Brown corpus le cd

Dotplot of the matrix from Table which shows word rep etitions in the

phrase to be or not to be

The dotplot of four concatenated Wal l Street Journal articles

Graphical illustration of the working of the optimization algorithm

Graphical illustration of the working of the optimization algorithm after one

b oundary has b een identied

The rst outside density plot of four concatenated Wal l Street Journal articles

Twoways to handle ignored words On the left dotplot ignored words may

not participate in matches On the dotplot on the right ignored words have

b een eliminated

The p erformance of the b est p erforming version of each algorithm A p erfect

score would b e exact matchesat distance on the graph

Example from the HUB corpus showing the annotation of topic segments

pro duced by the LDC Vertical whitespace is used to indicate changes in

sp eaker or background recording condition

PrecisionRecall curve for algorithms ME and WF when tested on the

HUB news broadcast Data The lone p ointat represents base

line p erformance

PrecisionRecall curve for algorithm WF on data from the HUB

Corpus when all words in the corpus were treated as unknown Performance

of the original WF algorithm is shown for comparison

PrecisionRecall curve for HUB news broadcast data Performance is

shown for algorithm WF on sp eechrecognized data and manual transcrip

tions of the same data Baseline p erformance is also shown xvii

PrecisionRecall curve for Spanish news broadcast Data from the HUB

Corpus

Diagram of the HMMs Mittendorf and Schauble used for passage retrieval

The transition probabilities p and p were b oth set to

Example of a segment identied by ME whichwould b e returned to a user

of an IR system The whitespace indicates where an additional b oundary

was placed by the annotators

Sample completed templates from an information extraction task

Example text indicating how topic segments could b e useful for information

extraction A management changeover template should be lled from the

sentence in b old xviii

Chapter

Intro duction

At the end of the twentieth century p eople contend with an everincreasing quantity

of information in various forms from numerous sources Radio television the internet

consumer electronic devices and other people bombard us daily with information ab out

many sub jects in many ways Much of this information is contained within what we will

call documents for lack of a b etter term By a do cument we mean a rep ository for a

snipp et of natural language in any medium whic h can be accessed frequently using a

computer after it is created Recorded radio broadcasts and television programs b o oks

pap ers stored on a computer handwritten notes scanned rep orts web pages voice mails

and faxes are all examples of do cuments Unrecorded conversations for example would

not b e considered do cuments according to our denition b ecause they lack p ersistence

The number of do cuments in existence to day is staggering The database service

LexisNexis alone indexes more than billion of them and their collection grows by

million per week LexisNexis The number of pages on the worldwide web

users of the p opular search engine Alta Vista can WWW is dicult to count but

search on the order of a terabyte of data and Alta Vista do es not index the entire web

Digital Equipment Corp oration The Library of Congress collection numb ers over

million items Even though many of these items such as photographs are not do c

uments according to our denition this is undoubtedly one of the largest collections of

do cuments ever assembled Library of Congress

Do cuments are generated at a far greater rate to day than ever b efore Despite the

claims ab out the imp ending pap erless oce made in the s and earlier Lancaster

the increased pace is not due exclusively to the growth of broadcast industries or the

rise of the internet The rate of pro duction of pap er do cuments is growing as evi

denced by the fact that worldwide pap er pro duction doubled between and

Fo o d and Agriculture Organization of the United Nations In fact instead of pro

ducing fewer pap er do cuments to day b ecause of information technologywe are pro ducing

more b ecause it is easier to do sohigh qualityprinters are inexp ensive and word pro cess

ing and desktop publishing packages are ubiquitous

The proliferation of do cuments due primarily to increased access to information tech

nology is unlikely to cease and will most likely b ecome an even larger burden on the

average consumer of information until the technology to access a desired piece of informa

tion catches up with the technology to create new do cuments Fox et al Since we

are unlikely to stem the tide of do cuments any time so on we need b etter ways to cop e with

themthat is to store them index them access them convert them from one form into

another more easily used form and so forth These needs are partly resp onsible for inter

est in research areas as seemingly diverse as digital libraries optical character recognition

OCR sp eech recognition information extraction and information retrieval IR

The common thread running through these lines of research is access to needed informa

tion This is obviously the purp ose of b oth information retrieval and information extrac

tion The goal of information retrieval is to p ermit do cument databases to b e searched in

arbitrary ways Information extraction fo cuses on identifying predetermined typ es of infor

mation as new do cuments are entered into a do cument collection Digital libraries provide

enhanced access to collections by making electronic do cuments accessible at a distance and

at least in theory p ermitting more do cuments to b e stored in a single rep ository They also

allow users to searchinways traditional libraries do not OCR and sp eech recognition b oth

enhance access b y converting do cuments from one form which is dicult to manipulate

with to days access to ols into another form namely text the medium required by most

natural language pro cessing information retrieval and information extraction systems

Motivation

This dissertation is ab out techniques for improving access to information and improving the

underlying language technology that helps provide access by dividing lengthy do cuments

into topicallycoherent sections Suchtechnology is necessary b ecause p eople who search

for information are interested in nding it in convenientlysized packages Clearly it is

unhelpful to learn that the answer to a question can be found in the lo cal libraryor for

that matter on the internet or somewhere in a Dow Jones database It is marginally more

useful to discover that the answer lies in the section of b o oks whose call numb ers are in the

range from to but even at the average community library lo oking through all these

titles let alone reading through each b o ok would require to o much time Even nding

out that the needed information was in a particular b o ok might not b e suciently sp ecic

if the information was needed quickly or in the event that more time were available if

reading through the less immediately relevant material in the b o ok would not b e b enecial

Generally the b est p ossible solution to a request for information is to give the requester

the answer if there is one Often times however there is not strictly sp eaking an answer

Many queries posed to IR systems are op enended enough that even if technology were

much more advanced than it is to day the best resp onse would still consist of a set of

do cuments Consider the query Who shot John F Kennedy The answer Lee Harvey

Oswald would suce in some cases but alternate theories ab ound Ideally the user of an

IR system should b e made aware of some of them

T echnology is not yet advanced enough to provide a simple answer likeLee Harvey

Oswald for many queries The answer to this particular query may b e found in a database

ab out the presidents but many information retrieval systems search large text databases

and extracting answers to arbitrary natural language questions from text is an op en re

search area A good compromise given the state of technology is to limit the size of the

do cuments presented to information seekers to b e just large enough to satisfy their infor

mation needs but no larger No one should b e exp ected to read the Warren Commission

rep ort in its entiretytolearnthatOswald was almost certainly JFKs assassin

Another p otential approach to satisfying demands for information is to synthesize mate

rial from selected passages within a do cument or even across multiple do cuments thereby

creating new do cuments ondemand which present the requested information in summary

form There is currently a variety of work b eing done in this area but we will touch on

this topic only briey See for instance Aone et al

There are imp ortant constraints on what just large enough should mean when re

sp onding to a demand for information It is crucial that the returned material be inter

pretable There should b e few unresolved pronominal references sucient context to allow

ambiguous words and phrases to b e understo o d and enough information to resp ond to the

original query If these constraints were unimp ortant then arbitrary passages p ossibly

b eginning and ending midsentence could b e presented to users IR systems could index

all do cument fragments consisting of con tiguous sequences of words and return them to

users in resp onse to their queries Although this can place an unb earable storage burden on

systems which index large collections IR p erformance do es improve using this technique

See Callan Kaszkiel and Zob el among others

A mo dest improvement to this approach is to limit the indexed segments to those

b eginning at sentence b oundaries No resp onse presented to a user would then b egin

with an uninterpretable sentence fragment This would also reduce the amount of storage

space required by the IR system However problems still remain Pronouns may not

be resolvable and there may not b e enough context to understand the meaning of a text

fragment

A b etter solution would b e to divide do cuments into sections with each section limited

to the extent of a particular topic and the b oundaries b etween topic segments aligned with

sentence b oundaries for clarity This would result in a minimal increase in the storage

space required by an IR system and would solve many of the problems stemming from lack

of context Ideally pronominal references to entities outside the section would b e resolved

so that sections were as intelligible on their own as p ossible

The diculty of making these sections stand on their own varies greatly and dep ends

on the typ e of do cument being partitioned Some typ es of do cuments are created with

partitions in place Newspap ers are divided into many articles and each article is lab eled

with a title Readers can cho ose to read or not to read articles based on their titles

of interviews Do cuments without any lab els or segment b oundaries such as transcripts

telephone conversations and other sp ontaneous communications lie at the opp osite end

of the continuum It might seem that no additional partitioning of lab eled do cuments is

necessary but the level of sub division is rarely ne enough Short newspap er articles may

b e restricted to a single topic but feature articles and articles from magazines and journals

may range over a numb er of related topics Do cuments in other media mayhave segments

marked implicitly but recovering the b oundaries b etween the segments may b e dicult

This diculty p ertains to segmenting television and radio news broadcasts which are

pro duced with the indep endence of stories in mind It is obvious to viewers of news

programs where the b oundaries b etween stories o ccur But it can be challenging to nd

the p oint where one story ends and another b egins using only a transcript

Segmentation Cues

In this dissertation we fo cus on segmenting do cuments using cues present in text and

transcrib ed sp eech There are however two additional sources of cues for segmenting video

broadcasts one of which applies equally well to audio data The video p ortion of programs

can be a rich source of information ab out transitions For example all black frames

sometimes app ear b etween stories and shifts from onlo cation rep orts back to the anchor

in the studio often result in more dramatic alterations of image content than are caused by

camera movement or shifting the fo cus of attention from one anchor to another Cues like

these have b een used to divide video do cuments into scenes eg Yeo and Liu

The second source of cues is the sp eech stream itself Some linguistic cues present in

sp eech suchasintonation and the use of pauses are not generally transcrib ed Hirschberg

and Grosz studied the relationship b etween intonation and discourse structure and found

correlations between intonational features and annotators lab eling of discourse struc

ture Hirschb erg and Grosz Grosz and Hirschb erg Hirschb erg and Nakatani

showed that annotators more consistently segment discourse using sp eech in conjunction

with transcripts than using text alone and that annotators can reliably segment both

sp ontaneous and planned sp eech Hirschb erg and Nakatani Passoneau and Litman

studied interannotator consistency and the correlation between linguistic cues and seg

mentation They prop osed segmentation algorithms based on cue words referential noun

phrases and the presence of pauses The p erformance of their algorithms was encouraging

but not as go o d as human p erformance Passoneau and Litman

Ultimately textual sp oken and video cues should b e combined and incorp orated into a

videoondemand system like the Informedia Digital Video Library which p ermits video

do cuments to be searched and browsed in the same ways that information retrieval sys

tems do for textual do cuments Christel et al Hauptmann and Witbro ck We

leave the combination of textual cues with those from other sources to future work

The Noisy Channel Mo del

If we assume there is a canonical topic segmentation we can treat the creation of do c

uments without explicitly marked topic b oundaries as an instance of the noisy channel

mo del Cover and Thomas Use of this mo del has resulted in advances in statis

tical techniques for sp eech recognition Bahl et al and for many natural language

pro cessing tasks Brown et al a Brown et al b

We will use the pro duction of a news broadcast as an example of how the noisy channel

mo del applies The pro ducer of a news program b egins with a collection of indep endent

stories The set of stories she intends to convey corresp onds to the Original Message in

Figure The content of the stories passes through an Enco der namely the journalists

who translate the content of the stories into words At this stage assuming each story is

written indep endently the b oundaries b etween stories are still clear We can consider the

written stories which will ultimately form the news broadcast to b e the Enco ded Message in

the gure This message passes through a Noisy Channel which according to a sto chastic

pro cess blurs or removes the b oundaries b etween stories There is a simple explanation for

this in the pro duction pro cess The goal of making the broadcast ow smo othly motivates

journalists to relate stories to one another and provide natural transitions b etween them

If every story b egan with a set phrase suchas In this story detecting b oundaries b etween stories would

b e trivial if this phrase was used nowhere else but news broadcasts would b e less captivating than they

currently are OriginalEncoded Corrupted Message Message Message Encoder Noisy

Channel

Figure The noisy channel mo del

These connections and transitions obscure the b oundaries b etween stories Once they are

in place the broadcast is in its nal form which corresp onds to the Corrupted Message

in the gure In broadcast form some topic b oundaries may still be marked by phrases

such as Coming up intonational cues or changes in image content In fact all of the

b oundaries may still b e detectable without determining the meaning of the program but

the markings will no longer be as explicit as they were b efore the stories passed through

the noisy channel

The ab ove example dealt only with the b oundaries b etween stories A similar pro cess

could obscure the topic b oundaries within individual stories from news broadcasts and

do cuments of other typ es as well In fact authors fo cus on making b oundaries subtle

since readabilityispartlya function of the smo othness of the transitions b etween topics

This can be said of participants in conversations as well but we fo cus on structuring

primarily monologues

Goals

We will describ e metho ds for recognizing the b oundaries between topic segments within

do cuments The best of our metho ds is applicable to a wide variety of do cument typ es

ts must be converted to text for in various media However the contents of the do cumen

these techniques to b e used

Until recently no topicb oundary annotated corp ora were available As a result we

will rst present our earliest results for simulations using concatenated do cuments In the

rest of this dissertation however we will demonstrate the eectiveness of our metho ds on

newspap er articles b oth transcrib ed and sp eechrecognized television and radio broadcasts

and works of literature We will also show that our metho ds apply to languages other than

English using transcripts of Spanish television and radio broadcasts

Evaluating the segmentations pro duced by text structuring algorithms presents several

problems Therefore we will discuss the merits of various p erformance measures and in

order to demonstrate the utility of the algorithms we present we will also describ e a number

of applications whichwould b enet from using topicsegmented do cuments For example

p erformance on various natural language pro cessing tasks such as coreference resolution

and wordsense disambiguation is likely to b e improved We will also describ e information

retrieval exp eriments using both the segmentations pro duced by human annotators and

those identied algorithmically

Topic Segmentation not Discourse Segmentation

The algorithms we prop ose for text segmentation are intended to be useful for language

engineering applications As a result they identify b oundaries which demarcate units of

text that are appropriately sized for these particular tasks The segments the b oundaries

delimit range in length from several sentences to several paragraphs Most theories of dis

course structure Grosz and Sidner Grosz et al Mann and Thompson

fo cus on relating much smaller units of discourse namely utterances As a result empiri

cal work on discourse segmentation has fo cussed primarily on identifying relations b etween

these units as well Hirschb erg and Grosz Passoneau and Litman Research

regarding the relationships between utterances has numerous implications for language

pro cessing tasks but is outside the scop e of this work

The coarsegrained segmentation we are pursuing may b e a subset of the more detailed

analysis that is the goal of discourse segmentation but we do not believe the techniques

features we use for topic seg we prop ose are applicable to discourse segmentation The

mentation do not have enough resolving p ower to b e of much use on a ner scale

For varietywewillfreelyinterchange the phrases text structuring text segmentation and topic segmen

tation We reserve discourse segmentation for predominantly hierarchical analyses that are nergrained

than those pro duced by the algorithms we present

Outline

In Chapter we will briey summarize theories of discourse structure that relate to text

structuring Knowledge of some of these theories is necessary to understand the algorithms

that we review in Chapter while others have inuenced the design of the algorithms we

describ e in Chapter We will then discuss features useful for identifying topic structure in

Chapter Some of these clues have b een used in previous approaches to text structuring

while others are novel Chapter reviews previous computational approaches and in

Chapter we will outline our own algorithms In Chapter we will describ e metho ds of

evaluating text structuring techniques and will measure the p erformance of our algorithms

using some of them In Chapter we will describ e applications which b enet from text

structuring and demonstrate the utility of text structuring techniques for a subset of these

applications Finallywe will summarize and discuss our ndings and outline some future

work in Chapter

Chapter

Theories of Discourse Structure

There is a large body of literature on the structure of discourse This work addresses a

wide range of issues including How large are the units whose relationships with one an

other should b e describ ed by a theory of discourse structure Harris Longacre

Stark What is the nature of the relationships b etween individual phrases in writ

ten texts Mann and Thompson What is the relationship b etween coherence atten

tion and the selection of referring expressions Grosz and Sidner Grosz et al

Is discourse structured hierarchically linearly or otherwise Hurtig Hinds

Skoro cho dko Webb er Sibun What are the relationships b etween the

utterances in multiparty discourse Hobbs Walker and Whittaker How can

largescale shifts in narrative texts b e identied Grimes Youmans

Since we cannot review the entire discourse structure literature we will summarize

only the articles most p ertinent to our work here The algorithms we present in Chapter

structure text in a theoryneutral way However they dep end implicitly on particular

answers to the ab ove questions

The algorithms we prop ose divide texts into sections which range from several sentences

to several paragraphs in length Much of the literature ab out discourse structure however

fo cuses on the relations between utterances and is only tangentially related to analyses

involving much larger fragments of discourse

Although there is ample evidence suggesting that discourse is hierarchically structured

Grosz and Sidner Webb er the algorithms we prop ose partition text linearly

The segments our metho ds identify contain utterances whose interrelations may b est be

describ ed by a hierarchical structure Identifying that structure is in the provenance of

discourse segmentation research however It may even be that the segments our algo

rithms lo cate t naturally into a hierarchical structure Segmenting do cuments linearly

also facilitates comparison with algorithms from the literature on topic segmentation and

p ermits evaluation using available linearlysegmented corp ora

Some of the corp ora we use to evaluate text segmentation algorithms contain brief pas

sages of dialogue Many news broadcasts are hosted by several anchors who o ccasionally

another However the ma jority of such do cuments is mono discuss the news with one

logue As a result wedonotsurvey work on dialogue and reserve the application of our

techniques to more typical conversational text for future work

Belowwe review Halliday and Hasans theory of the typ es of relations b etween textual

elements which provide coherence to a do cument Their work inuenced the design of

several text segmentation algorithms including our own We review Grosz and Sidners

theory of discourse structure b ecause understanding one of the algorithms we will review

requires knowledge of it Finallywe summarize Skoro cho dkos theory of text organization

which underpinned Hearsts work on dividing long do cuments into subtopic sections

Halliday and Hasan

In their book Cohesion in English Halliday and Hassan describ e texture as a prop erty

p ossessed by a text but which an arbitrary collection of sentences do es not have Readers

can frequently tell whether or not a series of sentences exhibits texture The sentences in

Example do exhibit it while those in Example do not Halliday and Hasan

Wash and core six co oking apples Put them into a repro of dish

Wash and core six co oking apples The prices of computers drop regularly

Cohesion is one of the elements of a discourse whichcontributes to its texture Cohesion

is present when an element in a text is best interpreted in light of a previous or less

frequently following element of the same text Halliday and Hasan identify ve cohesive

relations whichcontribute texture to a do cument

Reference References are likepointers Rather than rep eat a phrase in the text a writer

or sp eaker mayuseapointer to the entity selected by a phrase instead Halliday and

Hasan distinguish two main typ es of reference Exophoric references are to entities

in the world of the discourse and endophoric references are to p ortions of the text

itself The word he in Example is an exophoric reference and so is an endophoric

reference in Example

John likes apples but he loves p ears

For hes a jolly go o d fellow And so say all of us

Substitution Substitution and reference are similar but dier in that substitution o ccurs

prior to semantic interpretation while reference o ccurs after interpretation That is

a substitute acts merely as a pointer to aregionoftext which refers to an entity in

the world of the discourse while a reference refers directly to an entity without the

mediation of the original referring phrase In Example does substitutes for the

phrase like apples

Do you like apples Everyb o dy do es

Ellipsis Ellipsis is similar to substitution It can b e viewed as substitution by a zero In

Example bought has b een replaced byanull phrase in Mary some owers

John b ought some cho colates and Mary some owers

Conjunction Conjunction is more dicult to dene than the previous three relations It

holds b etween elements of a text when they are ordered temp orally one causes the

other when they describ e a contrast or when one elab orates on the other Examples

from Cohesion in English will demonstrate these relations Each of the sentences a

through d should b e read immediately following the rst sentence in Example

For the whole day he climb ed up the mountainside almost without stopping

a Then as dusk fell he sat down to rest Temp oral order

b So by night time the valley was far b elowhim Causation

c Yet he was hardly aware of b eing tired Contrast

d And in all this time he met no one Elab oration

Lexical cohesion Lexical cohesion holds b etween twotokens in a text which are either of

the same typ e or are semantically related in a particular way There are ve semantic

relations that constitute lexical cohesion

Reiteration with identity of reference o ccurs when a particular entity previously

referred to in a discourse is referred to again

John saw a dog

The dog was a retriever

Example refers to a particular dog and Example refers to the same dog

again

Reiteration without identity of reference o ccurs when reference is made to the

entire class to whichanentity previously referred to in a discourse b elongs

John saw a small retriever

Retrievers are usually large

Example refers to one particular member of the set of dogs identied as

retrievers while Example refers to the entire class of retrievers

Reiteration by means of superordinate occurs when reference is made to a su

p erclass of the class to which a previously mentioned entity b elongs

John saw the retriever

Dogs are his favorite animals

Example refers to a retriever whichisa typ e of dog while Example

refers to dogs in general

A systematic semantic relation holds when a word or group of words has

a clearly denable relationship with a previously used word or phrase For

example b oth could refer to memb ers of the same set

John likes retrievers

He do esnt like collies

Example refers to retrievers and Example mentions collies b oth of

which are subsets of the sp ecies of dogs In this case the relationship can be

classied as memb ership in a particular class

A nonsystematic semantic relation holds b etween twowords or phrases in a dis

course when they p ertain to a particular theme or topic but the nature of their

relationship is dicult to sp ecify Recognizing this category in a computational

system would b e more dicult than recognizing the other categories

John sp ent the afterno on studying in his dormitory ro om

He loves attending college

A semantic connection exists b etween the word dormitory in Example and

col lege in Example but it is hard to classify and unlikely that all such rela

tions or even the prep onderance of them could b e found in a knowledge source

in the waythatmanysynonymy relations can b e identied using a thesaurus

Halliday and Hasans categories overlap to some degree For example it can b e dicult

to distinguish instances of substitution from endophoric reference Substitution is subtly

dierent in that it relates words of the text is not a semantic relation and requires the

substituted phrase to have the same role as the phrase it substitutes for This is not the

case with reference Nonetheless Halliday and Hasan acknowledge that there are instances

where more than one category applies equally well

Halliday and Hasan explain that texts frequently exhibit varying degrees of cohesion in

dierent sections Obviously the start of a text cannot b e cohesive with preceding sections

nor can the end exhibit cohesion with later sections In the middle of a text however the

quantity of cohesion can vary greatly Some authors Halliday and Hasan suggest prefer

to alternate b etween high and low degrees of cohesion

Texturewhich is more frequently called coherenceand cohesion are often confused

but dier signicantly Cohesion relates elements of a text and can generally b e identied

See Hobbs Hobbs for an extended discussion of the dierence b etween the two

out of context Texture however is a prop ertythat applies to an entire text It is more

dicult to dene but can b e recognized up on reading a text in its entirety

Grosz Sidner

In their inuential pap er on the organization of discourse Grosz and Sidner Grosz

and Sidner present a tripartite theory Their theory addresses the relationships b etween

attentional state linguistic structure and intentional structure

Attentional state p ertains to conversants fo cus of attention and the accessibility and

salience of discourse entities Grosz and Sidner track attentional state using a stack

and the standard stack op erators push and pop The elements of the stack are data

structures called focus spaces which contain lists of available entities and the discourse

segment purpose which is an element of the intentional structure discussed below The

top element of the stack the fo cus space most recently added and some of the elements

lower in the stack are available to the hearer or reader of an utterance Asp ects of the

linguistic structure determine how fo cus spaces are added to and removed from the stack

The linguistic structure captures the relationships b etween successive utterances and

divides a text into discourse segments These segments form a hierarchical structure

The linguistic structure constrains changes in attentional state The fo cus space stack is

up dated when transitioning from one segment to another When leaving a segment and

entering a sister segment the stack is p opp ed prior to pushing the fo cus space asso ciated

with the new segmentonto it When entering an emb edded segment the fo cus information

asso ciated with that segment is simply pushed onto the stack

The intentional structure mo dels the goals and subgoals of the discussants The dis

course segment purp ose DSP is the intention asso ciated with a discourse segment Grosz

and Sidner call the intention of the entire discourse the disc ourse purpose DP They

classify the relationships between intentions which can be identied from the linguistic

structure as either dominance or satisfaction precedence Dominance means that satis

fying the dominated intention contributes to the satisfaction of the dominating intention

Satisfaction precedence indicates that one intention cannot be satised until another is

itself satised These relations mirror relations in the linguistic structure When one

intention dominates another in the intentional structure then the dominated intention

corresp onds to a discourse segment in the linguistic structure which is a descendantofthe

discourse segment related to the dominating intention A satisfactionprecedence relation

ship b etween twointentions indicates that the corresp onding discourse segments are sisters

in the linguistic structure

An example similar to one from Grosz and Sidners pap er will illustrate the rela

tionships between the comp onents of this theory Imagine a text consisting of only four

sentences each of which constitutes a separate discourse segment The rst sentence is

ab out a particular topic which the second and third sentences supp ort The nal sen tence

is ab out a dierent sub ject Figure shows the linguistic structure in terms of discourse

segments the intentional structure as enco ded by the list of dominance relations and the

attentional structures represented by a stackcontaining fo cus spaces Part of the gure

represents the state of the mo del after the rst two sentences have b een pro cessed Once

the third sentence of the discourse has been pro cessed the structures would be up dated

to corresp ond to those shown in part of the gure After the nal sentence has been

pro cessed the structures would b e as shown in part

In part the rst sentence lab eled discourse segment DS has contributed the

b ottom element to the fo cus space stack while the second sentence DS has contributed

the top element The DSP asso ciated with the rst segment DSP dominates the DSP

asso ciated with the second segment DSP When the third sentence is pro cessed the

hearer recognizes that it is in a separate discourse segment DS which is a sister to DS

This fact is reected in the linguistic structure both DS and DS are dominated by

DS which is indicated on the gure by indenting the dominated segments The mo del

of attentional state is up dated by pushing the fo cus space asso ciated with DS onto the

stack Pro cessing the fourth sentence do es not change the dominance hierarchy but do es

aect the attentional state The fo cus spaces asso ciated with b oth DS and DS would b e

p opp ed from the stack and the fo cus space asso ciated with DS would b e pushed Discourse segments Focus Space Stack Dominance Hierarchy

1 DS1 DS2 DSP 1 dominates DSP 2 Focus space 2

Focus space 1

2 DS1 DS2 DSP 1 dominates DSP 3 DS3 Focus space 3 DSP 1 dominates DSP 2

Focus space 1

3 DS1 DS2 DSP 1 dominates DSP 3 DS3 Focus space 4 DSP 1 dominates DSP 2

DS4

Figure Changes in the attentional intentional and linguistic structure when pro cessing

a sample text

Skoro cho dko

Skoro cho dko discussed metho ds of automatically generating abstracts for do cuments He

also outlined a typ ology of texts based on the presence of semantic relationships b etween

textual elementseither sentences or paragraphsin a do cument which he suggested

could b e identied using a semantic network Skoro cho dko He prop osed four typ es

of text

Chained Only neighb oring elements are strongly related

Ringed When the relationships b etween segments are represented as a graph this typ e

of text has a single cycle which encompasses all textual elements A newspap er

article with a leading summary some explanatory text and a conclusion is likely

to p ossess this typ e of structure since the rst and last segments would be related

and presumably neighb oring segments throughout the do cument would be related

as well Chained 12345

Ringed 12345

Monolithic

1 3

2 6

Piecewise Monolithic

1 3 5 7

Figure Skoro cho dkos four text typ es

Monolithic All of the segments of the text are related The graph for a text with this

kind of structure is a clique

Piecewise Monolithic Portions of the text are themselves monolithic but there are few

connections b etween the monolithic p ortions

Figure contains diagrams representing the four typ es of structure Skoro cho dko

identied Circles represent textual elements and lines indicate the presence of semantic

relations b etween them The numb ers asso ciated with textual elements indicate their order

in the text The gure shows that do cuments having the chained structure could b e easily

segmented Those exhibiting the ringed structure could be segmented as well If the

relationship b etween the rst and nal units is ignored a do cument with ringed structure

is transformed intoonewithc hained structure Piecewise monolithic do cuments could also

be segmented straightforwardly In fact these texts could be decomp osed into segments

ab out individual topics which are themselves monolithic in nature Monolithic texts p ose

a problem for text segmentation techniques since all their elements are related

Chapter

Structuring Clues

Text structuring algorithms from the literature employ a wide variety of features In this

chapter we will describ e most of these features as well as those used by our algorithms

We will describ e features used by only a single system from the literature in Chapter

in conjunction with the algorithms that employ them We will rst summarize some

interesting clues suggested in the literature which cannot b e used in computational systems

given current natural language pro cessing technology These cues highlight the fact that

people may use dierent cues to discover topic shifts than computers are currently capable

of using At the end of this chapter we will list some additional indicators of topic structure

which could b e used by future topic segmentation systems

Currently Uncomputable Features

Do cuments exhibit a number of features whichhave b een describ ed as b eing strong indi

cators of topic shifts However some of these features cannot easily be recognized using

current computational techniques We survey the most interesting of these features b elow

Grimes

Grimes suggested four clues ab out the presence of topic b oundaries The rst three indi

cators are found only in narratives Shifts to new segments maybemarked bychanges to

the

Scene

Participant orientation

Chronology

Theme

These indicators do not always mark topic shifts and a particular b oundary mayhave

multiple markers Further explanation of these indicators is in order First a change in

the setting of the action as is frequently found in novels and plays often indicates a topic

shift Next changes in participant orientationthe imp ortance ranking of characters in

a storycan aect the storys structure For instance a new topic segment often b egins

when a new character is intro duced and the fo cus of attention shifts away from previously

imp ortant characters A shift from one time period to anotherfor example from the

evening of one dayto the morning of the next without any description of the interv ening

timemost likely indicates the start of a new segment Finally shifts in theme which

can b e conveyed through dialogue and whichfrequently o ccur indep endentof changes in

setting and time p erio d may signal the b eginning of a new topic segment Grimes

Nakhimovsky

Nakhimovsky identied four typ es of shift from one text segment to another in narratives

These are

Topic shifts

Shifts in space and time

Discontinuities of gure and ground

Changes in narrative p ersp ective

Nakhimovsky also devised heuristics for segmenting narrative discourse but most of

these would be dicult to implement Nakhimovsky His heuristics recognize the

four typ es of shift he identied A c hange in topic may be signaled when there are no

anaphoric relations between a new segment and an info cus segment and there are no

obvious inferences which relate the new segment to an info cus segment Nakhimovsky

himself acknowledged that this was vague unless the inferences were dened with reference

to the abilities of a particular computational system Temp oral discontinuities are marked

by tense shifts or ashbacks to earlier scenes Shifts in gure and ground are accompanied

by asp ectual changes or changes in the discourse fo cus from temp oral to spatial informa

tion A change in narrative p ersp ective may be accompanied by among other things a

shift in story teller Shifting from an uninvolved narrator to a particular character who

relates the action in the rst p erson is one example of changing narrative p ersp ective

Summary

The techniques prop osed by Grimes and Nakhimovsky should b e straightforward for p eople

to use when pro cessing discourse However they are currently dicult to implement

b ecause they require a level of natural language understanding which cannot b e attained

using current technology For example how could ashbacks b e recognized The features

in the next section require much shallower pro cessing Although they would be more

tedious for p eople to use to manually identify topic b oundaries they can b e implemented

in an automatic text segmentation system

Currently Computable Features

In this section we describ e the features used by topic segmentation systems from the lit

erature and those used by the algorithms we describ e in Chapter We present examples

of these features using sample texts from the HUB Broadcast News Corpus collected by

the Linguistic Data Consortium LDC for the sp ok en do cument retrieval p ortion of the

TREC Text Retrieval Conference HUB Program Committee Boundaries

between topics in these examples are generally indicated with vertical white space The

text in these examples was transcrib ed from sp eech and lacks prop er capitalization punc

tuation sentence breaks and paragraph b oundaries We pro duced the gures in which

words and phrases are linked using SRAs Discourse Tagging To ol

actually also although and basically

b ecause but essentially except nally

rst further generally however indeed

like lo ok next no now

ok or otherwise right say

second see similarly since so

then therefore well yes

Table Cue words from Hirschb erg and Litman

Cue Words Phrases

Grosz and Sidner explained that some words in discourse are used to indicate changes in

the discourse structure rather than to convey information ab out the sub ject matter b eing

discussed Their example is Incidental ly Jane swims every day Since it is improbable that

Jane unintentionally happ ens to swim dailythe most likely interpretation of incidental ly

is that it conveys information ab out the relationship of the utterance Jane swims every

day to the current discourse That is incidental ly marks a brief diversion from the main

topic and signals the start of a new discourse segment dominated by the current segment

Grosz and Sidner

Other researchers have examined the relationship b etween particular words and phrases

and discourse structure as well Reichman Dahlgren DiEugenio et al

Cue words play an imp ortant role in the discourse segmentation work of Hirschb erg and

Litman among others They presentanumber of cue words that indicate changes in dis

course structure which they gleaned from various sources Hirschb erg and Litman

They used these cue words which are shown in Table to study the corresp ondence

between intonation and cue word usage in sp eech These cue words are relatively domain

indep endent and may mark topic shifts in many genres

Domain Cues

The cue words and phrases weintend to use dier from those Hirschb erg and Litman used

in two imp ortant ways First they are highly domainsp ecic Domainsp ecicity means

that new lists must b e created b efore do cuments from new sources can b e segmented This

is a drawback b ecause manually creating the lists is timeconsuming and automatically

creating them requires an annotated corpus However the advantage is that genresp ecic

conventions are often reliable indicators of topic shifts For example in the domain of

news broadcasts cue phrases involving greetings suchas good evening good night and good

morning o ccur almost exclusively at the b eginning and end of broadcasts in brief segments

which b egin or end each show This makes them go o d indicators of shifts from one topic

segment to the next The cue phrase good evening is highly domain sp ecic In the Penn

Wal l Street Journal Treebank Marcus et al it do es not o ccur at all in million

words of text Even if it is found in other Wal l Street Journal articles there is no guarantee

that it will mark a topic shift

Second some of our cue phrases contain sequences of words of particular typ es suchas

p erson or place names The presence of word sequences of particular typ es makes the cue

phrases dynamic Another example from news broadcasts will illustrate this Rep orters

often conclude onlo cation rep orts with the phrase reporting from followed by the name of

a placea country or city for example Since these rep orts are often followed by new topic

segments this typ e of phrase is a go o d indicator of a topic b oundary Employing mildly

pro ductive cue phrases reduces the numb er of misidentications For example rather than

lab el only instances of Im followed by a rep orters name we could identify all instances

of Im and treat them as cue phrases Although simpler wewould mislab el many noncue

phrases this way We would mislab el Im in the phrase Im late for example as a cue

phrase

Toprevent confusion with more conventional notions of what a cue phrase is werefer

to the cue words and phrases weuseasdomain cues Wemanually identied our domain

cues from the broadcast news domain and separated them into several categories These

categories are new p erson cues greeting cues intro ductory cues pointers to up coming

stories shifts to other broadcasters returns from commercials and signingo cues New

p erson cues o ccur when guests join the broadcasters or rep orters are intro duced Greetings

usually mark the b eginning or end of a broadcast Intro ductory cues frequently accom

keep the pany the b eginning of news stories Pointers to up coming stories are used to

audience interested b ecause of the suggested imp ortance of what will be discussed later

in the broadcast The LDC eliminated commercials from the corpus we used but phrases

indicating that a commercial just ended remained and are useful indicators of topic shifts

The nal category signo cues contains all of the pro ductive phrases we used and signals

the transition from an onlo cation rep ort backtotheanchor in the studio

Table lists the sp ecic domain cues we used person is shorthand for a p ersons

name place refers to a lo cation and station indicates the call letters of a station or

networkfor example C N N Figure is an example from the HUB broadcast news

corpus which contains one of these cue phrases

We divided the domain cues into categories b ecause not all cues indicate up coming topic

shifts Greetings most often precede new topic segments but intro ductory phrases more

frequently follow topic b oundaries Our text segmentation algorithms use only the category

of each phrase b ecause individual phrases o ccurred to o rarely for them to b e used reliably

in a statistical mo del With more training data we could explore the appropriateness

of the categories we established and test the value of individual cue phrases as well In

Section weshow that with a larger corpus we can induce a list of useful domain cues

Most of the domain cue categories are selfexplanatory However the greeting category

contains some phrases unlikely to b e commonly regarded as greetings These are phrases

suchasbrought to you by and words like transcript which often o ccur at the start or end

of a broadcast and which rarely o ccur elsewhere As a result they b ehave likethe other

memb ers of the greeting category to indicate up coming topic shifts but are not obviously

greetings in the waythatgood morning is

Computing Domain Cues

We identied television station and network names using regular expressions To lab el

words with the categories person and place we built a maximum entropy mo del using

the mo deling to ols designed and implemented by Adwait Ratnaparkhi which have been

used for partofsp eech tagging parsing and endofsentence detection among other things

Ratnaparkhi Ratnaparkhi a Reynar and Ratnaparkhi

See Berger et al for an intro duction to maximum entropy mo deling for natural language pro

cessing and Ratnaparkhi b for information ab out Ratnaparkhis implementation in particular

Domain Cue Category

joining us New p erson

go o d night Greeting

good evening Greeting

go o d morning Greeting

hello Greeting

thats the news Greeting

tomorrowon Greeting

broughttoyou by Greeting

transcript Greeting

top story Intro ductory