Answering Questions from Multiple Documents – the Role of Multi-Document Summarization

Pinaki Bhaskar Department of Computer Science & Engineering, Jadavpur University, Kolkata – 700032, India [email protected]

Text Retrieval Conference (TREC)1 has spon- Abstract sored a question-answering track, which evalu- ates systems that answer factual questions by Ongoing research work on Question Answer- consulting the documents of the TREC corpus. A ing using multi-document summarization has number of systems in this evaluation have suc- been described. It has two main sub modules, cessfully combined and document retrieval and Multi-document Sum- natural language processing techniques. More marization. We first preprocess the documents recently, Conference and Labs of Evaluation Fo- and then index them using Nutch with NE rums (CLEF)2 are organizing QA lab from 2010. field. Stop are removed and NEs are 3 tagged from each question and all remaining INEX has also started question words are stemmed and then retrieve track. INEX 2011 designed a QA track (SanJuan the most relevant 10 documents. Now, docu- et al., 2011) to stimulate the research for real ment graph-based query focused multi- world application. The Question Answering document summarizer is used where question (QA) task is contextualizing tweets, i.e., answer- words are used as query. A document graph is ing questions of the form "what is this tweet constructed, where the nodes are sentences of about?" INEX 2012 Tweet Contextualization the documents and edge scores reflect the cor- (TC) track gives QA research a new direction by relation measure between the nodes. The sys- fusing IR and summarization with QA. The first tem clusters similar texts from the graph using this edge score. Each cluster gets a weight and task is to identify the most relevant document, has a cluster center. Next, question dependent for this a focused IR is needed. And the second weights are added to the corresponding cluster task is to extract most relevant passages from the score. Top two-ranked sentences of each clus- most relevant retrieved documents. So an auto- ter is identified in order and compressed and matic summarizer is needed. The general pur- then fused to a single sentence. The com- pose of the task involves tweet analysis, passage pressed and fused sentences are included into and/or XML elements retrieval and construction the output summary with a limit of 500 words, of the answer, more specifically, the summariza- which is presented as answer. The system is tion of the tweet topic. tested on data set of INEX QA track from Automatic text summarization (Jezek and 2011 to 2013 and best readability score was achieved. Steinberger, 2008) has become an important and timely tool for assisting and interpreting text in- 1 Introduction formation in today’s fast-growing information age. An Abstractive Summarization ((Hahn and With the explosion of information in Internet, Romacker, 2001) and (Erkan and Radev, 2004)) Natural language Question Answering (QA) is attempts to develop an understanding of the main recognized as a capability with great potential. concepts in a document and then expresses those Traditionally, QA has attracted many AI re- concepts in clear natural language. Extractive searchers, but most QA systems developed are Summaries (Kyoomarsi et al., 2008) are formu- toy systems or games confined to laboratories and to a very restricted domain. Several recent 1 conferences and workshops have focused on as- http://trec.nist.gov/ 2 http://www.clef-initiative.eu// pects of the QA research. Starting in 1999, the 3 https://inex.mmci.uni-saarland.de/

14 Proceedings of the Student Research Workshop associated with RANLP 2013, pages 14–21, Hissar, Bulgaria, 9-11 September 2013. lated by extracting key text segments (sentences sage clustering and then composes an extraction or passages) from the text, based on statistical summary, which reflects these main themes. analysis of individual or mixed surface level fea- Graph-based methods have been also proposed tures such as /phrase frequency, location or for generating summaries. A document graph- cue words to locate the sentences to be extracted. based query focused multi-document summariza- Our approach is based on Extractive Summariza- tion system has been described by Paladhi et al. tion. (2008) and Bhaskar and Bandyopadhyay (2010a In this paper, we describe a hybrid Question and 2010b). Answering system of document retrieval and In the present work, we have used the IR sys- multi-document summarization. The document tem as described by Pakray et al. (2010 and retrieval is based on Nutch4 architecture and the 2011) and Bhaskar et al. (2011) and the automat- multi-document summarization system is based ic summarization system as discussed by graph, cluster, sentence compression & fusion Bhaskar and Bandyopadhyay (2010a and 2010b) and sentence ordering. The same sentence scor- and Bhaskar et al. (2011). ing and ranking approach of Bhaskar and Ban- dyopadhyay (2010a and 2010b) has been fol- 3 System Architecture lowed. The proposed system was run on the data In this section the overview of the system set of three years of INEX QA track from 2011 framework of the current INEX system has been to 2013. shown. The current INEX system has two major 2 Related Work sub-systems; one is the Focused IR system and the other one is the Recent trend shows hybrid approach of question system. The Focused IR system has been devel- answering (QA) using Information Retrieval (IR) oped on the basic architecture of Nutch, which can improve the performance of the QA system. use the architecture of Lucene5. Nutch is an open Schiffman et al. (2007) successfully used meth- source , which supports only the ods of IR into QA system. Rodrigo et al. (2010) monolingual Information Retrieval in English, removed incorrect answers of QA system using etc. The Higher-level system architecture of the an IR engine. Pakray et al. (2010) used the IR combined Tweet Contextualization system of system into QA and Pakray et al. (2011) pro- Focused IR and Automatic Summarization is posed an efficient hybrid QA system using IR. shown in the Figure 1. Tombros and Sanderson (1998) presents an investigation into the utility of document summa- rization in the context of IR, more specifically in the application of so-called query-biased sum- maries: summaries customized to reflect the in- formation need expressed in a query. Employed in the retrieved document list displayed after re- trieval took place, the summaries’ utility was evaluated in a task-based environment by meas- uring users’ speed and accuracy in identifying relevant documents. A lot of research work has been done in the domain of both query dependent and independent summarization. MEAD (Radev et al., 2004) is a centroid based multi document summarizer, which generates summaries using cluster cen- troids produced by topic detection and tracking Figure 1. Higher-level system architecture system. NeATS (Lin and Hovy, 2002) selects important content using sentence position, term 4 Document Retrieval frequency, topic signature and term clustering. 4.1 Document and Indexing XDoX (Hardy et al., 2002) identifies the most salient themes within the document set by pas- The web documents are full of noises mixed with the original content. In that case it is very diffi-

4 http://nutch.apache.org/ 5 http://lucene.apache.org/

15 7 cult to identify and separate the noises from the standard Porter algorithm has been actual content. INEX 2012 corpus had some used to stem the question tokens. After stemming noise in the documents and the documents are in all the question tokens, queries are formed with XML tagged format. So, first of all, the docu- the stemmed question tokens. ments had to be preprocessed. The document structure is checked and reformatted according to 4.3 Document Retrieval the system requirements. After searching each query into the Lucene in- XML Parser: The corpus was in XML for- dex, a set of retrieved documents in ranked order mat. All the XML test data has been parsed be- for each question is received. fore indexing using our XML Parser. The XML First of all, all queries were fired with AND Parser extracts the Title of the document along operator. If at least ten documents are retrieved with the paragraphs. using the query with AND operator then the que- Noise Removal: The corpus has some noise as ry is removed from the query list and need not be well as some special symbols that are not neces- searched again. If not then the query is fired sary for our system. The list of noise symbols again with OR operator. OR searching retrieves and the special symbols like “"”, “&”, at least ten documents for each query. We always “'''”, multiple spaces etc. is initially developed ranked the retrieved document using AND opera- manually by looking at a number of documents tor higher than the same using OR operator. and then the list is used to automatically remove Now, the top ranked ten relevant documents for such symbols from the documents. each question is considered for milti-document Named Entity Recognizer (NER): After summarization. Document retrieval is the most cleaning the corpus, the named entity recognizer crucial part of this system. We take only the top identifies all the named entities (NE) in the doc- ranked ten relevant documents assuming that uments and tags them according to their types, these are the most relevant documents for the which are indexed during the document index- question from which the query had been generat- ing. ed. Document Indexing: After parsing the docu- ments, they are indexed using Lucene, an open 5 Multi-Document Summarization source indexer. 5.1 Graph-Based Clustered Model 4.2 Question Parsing The proposed graph-based multi-document After indexing has been done, the questions had summarization method consists of following to be processed to retrieve relevant documents. steps: Each question / topic was processed to identify (1) The document set D = {d1,d2, … d10} is the question words for submission to Lucene. processed to extract text fragments, which are The questions processing steps are described be- sentences in this system as it has been discussed low: earlier. Let for a document di, the sentences are Removal: In this step the question {si1, si2, … sim}. Each text fragment becomes a words are identified from the questions. The stop node of the graph. words6 and question words (what, when, where, (2) Next, edges are created between nodes which etc.) are removed from each question and across the documents where edge score repre- the words remaining in the questions after the sents the degree of correlation between inter- removal of such words are identified as the ques- documents nodes. tion tokens. (3) Seed nodes are extracted which identify Named Entity Recognizer (NER): After re- the relevant sentences within D and a search moving the stop words, the named entity recog- graph is built to reflect the semantic relationship nizer identifies all the named entities (NE) in the between the nodes. question and tags them according to their types, (4) Now, each node is assigned a question de- which are used during the scoring of the sentenc- pendent score and the search graph is expanded. es of the retrieved document. (5) A question dependent multi-document Stemming: Question tokens may appear in in- summary is generated from the search graph. flected forms in the questions. For English, Each sentence is represented as a node in the graph. The text in each document is split into

6 http://members.unine.ch/jacques.savoy/clef/ 7 http://tartarus.org/~martin/PorterStemmer/java.txt

16 sentences and each sentence is represented with a document graph as described by Bhaskar and vector of constituent words. If pair of related Bandyopadhyay (2010b). document is considered, then the inter document The construction of question independent part graph can be represented as a set of nodes in the of the Markov clusters completes the document- form of bipartite graph. The edges connect two based processing phase of the system. nodes corresponding to sentences from different documents. 5.3 Key Term Extraction Construct the Edge and Calculate Edge Key Term Extraction module has two sub mod- Score: The similarity between two nodes is ex- ules, i.e., question term extraction and Title pressed as the edge weight of the bipartite graph. words extraction. Two nodes are related if they share common Question Term Extraction: First the question words (except stop words) and the degree of rela- is parsed using the Question Parsing module. In tionship can be measured by equation 1 adapting this Question Parsing module, the Named Enti- some traditional IR formula (Varadarajan and ties (NE) are identified and tagged in the given Hristidis, 2006). question using the Stanford NER8 engine. The ∑ ((tf (t(u),w) + tf (t(v),w)) × idf (w)) remaining words after stop words removal are Edge_Score = w∈(t(u)∩t(v)) (1) stemmed using Porter Stemmer. size(t(u)) + size(t(v)) Title Word Extraction: The titles of each re- where, tf(d , w) is number of occurrence of w trieved documents are extracted and forwarded in d, idf (w) is the inverse of the number of doc- as input given to the Title Word Extraction mod- uments containing w, and size(d) is the size of ule. After removing all the stop words from the the documents in words. Actually for a particular titles, the remaining tile words are extracted and node, total edge score is defined as the sum of used as the keywords in this system. scores of all out going edges from that node. The nodes with higher total edge scores than some 5.4 Question Dependent Process predefined threshold are included as seed nodes. The nodes of the already constructed search But the challenge for multi-document summa- graph are given a question dependent score. Us- rization is that the information stored in different ing the combined scores of question independent documents inevitably overlap with each other. score and question dependent score, clusters are So, before inclusion of a particular node (sen- reordered and relevant sentences are collected tence), it has to be checked whether it is being from each cluster in order. Then each collected repeated or not. Two sentences are said to be sentence has processed and compressed remov- similar if they share for example, 70% words in ing the unimportant phrases. After that the com- common. pressed sentences are used to construct the sum- Construction of Search Graph: After identi- mary. fication of seed/topic nodes a search graph is Recalculate the Cluster Score: There are constructed. For nodes, pertaining to different three basic components in the sentence weight documents, edge scores are already calculated, like question terms, title words and synonyms of but for intra document nodes, edge scores are question terms dependent scores, which are cal- calculated in the similar fashion as said earlier. culated using equation 2. Since, highly dense graph leads to higher search / ! !! !!!! execution time, only the edges having edge � = !!! �! − � + 1 ! 1 − ×� (2) !! scores well above the threshold value might be where, w is the term dependent score of the considered. sentence i, t is the no. of the term, nt is the total ! 5.2 Identification of Sub-topics through no. of term, �! is the possession of the word Markov Clustering which was matched with the term t in the sen- tence i, Ni is the total no. of words in sentence i In this section, we will discuss the process to and b is boost factor of the term, which is 3, 2 or identify shared subtopics from related multi 1 for question terms, title words and synonyms source documents. We already discussed that the respectively. These three components are added subtopics shared by different news articles on to get the final weight of a sentence. same event form natural (separate) clusters of Recalculate the Cluster Ranking: We start sentences when they are represented using doc- by defining a function that attributes values to ument graph. We use Markov principle of graph clustering to identify those clusters from the 8 http://www-nlp.stanford.edu/ner/

17 the sentences as well as to the clusters. We refer 5.6 Sentence Compression to sentences indexed by i and question terms in- All the phrases which are in one of those 34 rela- dexed by j. We want to maximize the number of tions in the training file, whose probability to question term covered by selection of sentences: ! drop was 100% and also do not contain any �������� ! �! �! (3) question term, are removed from the selected ! where, �! is the weight of question term j in summary sentence as described by Bhaskar and the sentence i and qj is a binary variable indicat- Bandyopadhyay (2010a). Now the remaining ing the presence of that question term in the clus- phrases are identified from the parser output of ter. We also take the selection over title words the sentence and search phrases that contain at and synonyms of the question terms. We collect least one question term then those phrases are the list of synonyms of the each word in the selected. The selected phrases are combined to- questions from the WordNet 3.09. The general gether with the necessary phrases of the sentence sets of tile words and synonyms are indexed by k to construct a new compressed sentence for the and l respectively. So we also want to maximize summary. The necessary phrases are identified the number of title words and synonyms covered from the parse tree of the sentence. The phrases by a selection of sentences using similar calcula- with nsubj and the VP phrase related with the tion like question terms using equation 3. nsubj are some example of necessary phrases. So, the question dependent score of a cluster is the weighted sum of the question terms it con- 5.7 Sentence Selection for Summary tains. If clusters are indexed by x, the question The compressed sentences for summary have dependent score of the cluster x is: been taken until the length restriction of the ! ! ! summary is reached, i.e. until the following con- �! = �! �! dition holds: !!! ! � � < � (7) ! ! ! ! ! where, li is the length (in no. of words) of + �!� + � !� ! ! ! ! compressed sentence i, Si is a binary variable !!! ! !!! ! representing the selection of sentence i for the (4) ! summary and L (=100 words) is the maximum where, �! is the question dependent score of summary length. After taking the top two sen- the cluster x, n is the total no. of sentences in tences from all the clusters, if the length re- cluster x. Now, the new recalculated combined striction L is not reached, then the second itera- score of cluster x is: tion is started similar to the first iteration and the ! ! �! = �! + �! (5) next top most weighted sentence of each cluster where, cx is the new score of the cluster x and are taken in order of the clusters and compressed. g If after the completion of the second iteration cx is the question independent cluster score in same thing happens, then the next iteration will the graph of cluster x. Now, all the clusters are start in the same way and so on until the length ranked with their new score c . x restriction has been reached. 5.5 Retrieve Sentences for Summary 5.8 Sentence Ordering and Coherency Get the highest weighted two sentences of each In this paper, we will propose a scheme of order- cluster, by the following equation: ing; in that, it only takes into consideration the ! ! ! max ! �! �! + ! �!�! + ! �! �! ∀� (6) semantic closeness of information pieces (sen- where, i is the sentence index of a cluster. The tences) in deciding the ordering among them. original sentences in the documents are generally First, the starting sentence is identified which is very lengthy to place in the summary. So, we are the sentence with lowest positional ranking actually interested in a selection over phrases of among selected ones over the document set. Next sentence. After getting the top two sentences of a for any source node (sentence) we find the sum- cluster, they are split into multiple phrases. The mary node that is not already selected and have Stanford Parser10 is used to parse the sentences (correlation value) with the source node. This and get the phrases of the sentence. node will be selected as next source node in or- dering. This ordering process will continue until the nodes are totally ordered. The above ordering 9 http://wordnet.princeton.edu/ scheme will order the nodes independent of the 10 http://nlp.stanford.edu/software/lex-parser.shtml

18 actual ordering of nodes in the original docu- trashed, and readability of following passages ment, thus eliminating the source bias due to in- must be assessed as if these passages were not dividual writing style of human authors. Moreo- present, so they were marked as Trash (T). The ver, the scheme is logical because we select a readability evaluation scores are shown in the sentence for position p at output summary, based figure 3. Our relaxed metric i.e relevancy (T) on how coherent it is with the (p-1)th sentence. score is the best score and strict metric i.e aver- The main sentence’s number has been taken as age of non redundancy (R), soundness (A) and the sentence number of the fused sentence. syntax (S) score is the 4th best score among all Now the generated multi-document summary the runs from all the participants of INEX 2011. is presented as the answer of the corresponding question. 70 56.05 61.73 55.4 60 53.53 53.53 50.54 49.70 49.56 46.72 43.22 43.22 43.22 41.11 6 Experiment Result 50 40.90

40 34.79 The proposed system has been tested on the data set of INEX QA track from 2011 to 2013. 30 20 6.1 Informative Content Evaluation 10 The Informative Content evaluation (SanJuan et 0 al., 2011) by selecting relevant passages using 2011 2012 2013 simple log difference of equation 8 was used: Mean Average Relevancy (T) ⎛ max(P(t / reference),P(t / summary))⎞ Non redundancy (R) ∑log⎜ ⎟ (8) ⎝ min(P(t / reference),P(t / summary)) ⎠ Soundness (A) Syntax (S) The year wise evaluation scores of informa- Figure 3. The evaluation scores of Readability tiveness of all topics are shown in the figure 2. Evaluation

7 Discussion 1.5 The tweet question answering system has been 0.9984 0.9981 0.9973 0.9541 0.9397 0.9365 developed and tested on the data set of the Ques-

0.8481 Skip

1.0 0.7565 tion Answering (QA) / Tweet Contextualization 0.4841 (TC) track of the INEX evaluation campaign 0.5 unigram from 2011 to 2013. The overall system has been 0.0 evaluated using the evaluation metrics provided 2011 2012 2013 as part of the QA/TC track of INEX. Considering Figure 2. The evaluation scores of Informative- that the system is completely automatic and rule ness by organizers of all topics based and run on web documents, the evaluation results are satisfactory as readability scores are 6.2 Readability Evaluation very high and in the relaxed metric we got the For Readability evaluation (SanJuan et al., 2011) highest score of 43.22% in 2011, which will real- all passages in a summary have been evaluated ly encourage us to continue work on it in future. according to Syntax (S), Soundness/Anaphora Future works will be motivated towards im- (A), Redundancy (R) and Relevancy/Trash (T). proving the performance of the system by con- If a passage contains a syntactic problem (bad centrating on co-reference and anaphora resolu- segmentation for example) then it has been tion, multi-word identification, para-phrasing, marked as Syntax (S) error. If a passage contains feature selection etc. In future, we will also try to an unsolved anaphora then it has been marked as use , which will increase our Anaphora (A) error. If a passage contains any relevance score. redundant information, i.e., an information that have already been given in a previous passage Acknowledgments then it has been marked as Redundancy (R) er- We acknowledge the support of the DeitY, ror. If a passage does not make any sense in its MCIT, Govt. of India funded project “Develop- context (i.e., after reading the previous passages) ment of Cross Lingual Information Access then these passages must be considered as (CLIA) System Phase II”.

19 References tional ACM SIGIR conference on Research and development in information retrieval, pp. 121-128, Álvaro Rodrigo, Joaquın Pérez-Iglesias, Anselmo ISBN: 1-58113-561-0, ACM New York, NY, Peñas, Guillermo Garrido, and Lourdes Araujo. USA. 2010. A Question Answering System based on In- formation Retrieval and Validation, In: CLEF 2010 Karel Jezek, and Josef Steinberger. 2008. Automatic Workshop on Multiple Language Question An- Text summarization. In: Snasel, V. (ed.) Znalosti swering (MLQA 2010), Padua, Italy. 2008. ISBN 978-80-227-2827-0, pp.1--12. FIIT STU Brarislava, Ustav Informatiky a softveroveho Anastasios Tombros, and Mark Sanderson. 1998. Ad- inzinierstva. vantages of Query Biased Summaries in Infor- mation Retrieval. In: the 21st annual international Partha Pakray, Pinaki Bhaskar, Santanu Pal, Dipankar ACM SIGIR conference on Research and devel- Das, Sivaji Bandyopadhyay, and Alexander Gel- opment in information retrieval, pp. 2-10, ISBN:1- bukh. 2010. JU_CSE_TE: System Description 58113-015-5, ACM New York, USA. QA@CLEF 2010 – ResPubliQA. In: CLEF 2010 Workshop on Multiple Language Question An- Barry Schiffman, Kathleen McKeown, Ralph Grish- swering (MLQA 2010), Padua, Italy. man, and James Allan. 2007. Question Answering using Integrated Information Retrieval and Infor- Partha Pakray, Pinaki Bhaskar, Somnath Banerjee, mation Extraction. In: HLT/NAACL 07, pp. 532- Bidhan Chandra Pal, Alexander Gelbukh, and Si- 539, Rochester, NY, USA. vaji Bandyopadhyay. 2011. A Hybrid Question An- swering System based on Information Retrieval Bidhan Chandra Pal, Pinaki Bhaskar, and Sivaji Ban- and Answer Validation. In: Question Answering dyopadhyay. 2011. A Rule Base Approach for for Machine Reading Evaluation (QA4MRE) at Analysis of Comparative and Evaluative Question CLEF 2011, Amsterdam. in Tourism Domain. In: 6th Workshop on Knowledge and Reasoning for Answering Ques- Pinaki Bhaskar, and Sivaji Bandyopadhyay. 2010a. A tions (KRAQ'11) in IJCNLP 2011, pp 29-37, Thai- Query Focused Multi Document Automatic Sum- land. marization. In: the 24th Pacific Asia Conference on Language, Information and Computation (PACLIC Chin-Yew Lin, and Eduard Hovy. 2002. From Single 24), pp 545-554, Tohoku University, Sendai, Ja- to Multidocument Summarization: A Prototype pan. System and its Evaluation. In: ACL, pp. 457-464. Pinaki Bhaskar, and Sivaji Bandyopadhyay. 2010b. A Dragomir R. Radev, Hongyan Jing, Małgorzata Stys, Query Focused Automatic Multi Document Sum- Daniel Tam. 2004. Centroid- based summarization marizer. In: the International Conference on Natu- of multiple documents. In: Information Processing ral Language Processing (ICON), pp. 241--250. and Management. 40, pp. 919–938. IIT, Kharagpur, India. Eric SanJuan, V ́eronique Moriceau, Xavier Tannier, Pinaki Bhaskar, Amitava Das, Partha Pakray, and Patrice Bellot, and Josiane Mothe. 2011. Overview Sivaji Bandyopadhyay. 2010c. Theme Based Eng- of the INEX 2011 Question Answering Track lish and Bengali Ad-hoc Monolingual Information (QA@INEX). In: Focused Retrieval of Content and Retrieval in FIRE 2010. In: the Forum for Infor- Structure, 10th International Workshop of the Initi- mation Retrieval Evaluation (FIRE) – 2010, ative for the Evaluation of XML Retrieval (INEX), Gandhinagar, India. Geva, S., Kamps, J., Schenkel, R. (Eds.). Lecture Notes in Computer Sc., Springer. Pinaki Bhaskar, Somnath Banerjee, Snehasis Neogi, and Sivaji Bandyopadhyay. 2012a. A Hybrid QA Farshad Kyoomarsi, Hamid Khosravi, Esfandiar Es- System with Focused IR and Automatic Summari- lami, and Pooya Khosravyan Dehkordy. 2008. Op- zation for INEX 2011. In: Geva, S., Kamps, J., timizing Text Summarization Based on Fuzzy Log- Schenkel, R.(eds.): Focused Retrieval of Content ic. In: Seventh IEEE/ACIS International Confer- and Structure: 10th International Workshop of the ence on Computer and Information Science, pp. Initiative for the Evaluation of XML Retrieval, 347--352. IEEE, University of Shahid Bahonar INEX 2011. Lecture Notes in Computer Science, Kerman, UK. vol. 7424. Springer Verlag, Berlin, Heidelberg. Gu ̈ne ̧s Erkan, and Dragomir R. Radev. 2004. Pinaki Bhaskar, and Sivaji Bandyopadhyay. 2012b. LexRank: Graph-based Centrality as Salience in Answer Extraction of Comparative and Evaluative Text Summarization. In: Journal of Artificial Intel- Question in Tourism Domain. In: International ligence Research, vol. 22, pp. 457-479. Journal of Computer Science and Information Hilda Hardy, Nobuyuki Shimizu, Tomek Strzalkow- Technologies (IJCSIT), ISSN: 0975-9646, Vol. 3, ski, Liu Ting, G. Bowden Wise, and Xinyang Issue 4, pp. 4610 – 4616. Zhang. 2002. Cross-document summarization by Pinaki Bhaskar, and Sivaji Bandyopadhyay. 2012c. concept classification. In: the 25th annual interna- Cross Lingual Query Dependent Snippet Genera-

20 tion. In: International Journal of Computer Science Pinaki Bhaskar, Somnath Banerjee, and Sivaji Ban- and Information Technologies (IJCSIT), ISSN: dyopadhyay. 2013c. Tweet Contextualization (An- 0975-9646, Vol. 3, Issue 4, pp. 4603 – 4609. swering Tweet Question) – the Role of Multi- document Summarization. In: the Initiative for the Pinaki Bhaskar, and Sivaji Bandyopadhyay. 2012d. Evaluation of XML Retrieval, INEX 2013 at CLEF Language Independent Query Focused Snippet 2013 Conference and Labs of the Evaluation Fo- Generation. In: T. Catarci et al. (Eds.): Information rum, Valencia, Spain. Access Evaluation. Multilinguality, Multimodality, and Visual Analytics: Third International Confer- Pinaki Bhaskar, Somnath Banerjee, Partha Pakray, ence of the CLEF Initiative, CLEF 2012, Rome, It- Samadrita Banerjee, Sivaji Bandyopadhyay, and aly, Proceedings, Lecture Notes in Computer Sci- Alexander Gelbukh. 2013d. A Hybrid Question An- ence Volume 7488, pp 138-140, DOI 10.1007/978- swering System for Multiple Choice Question 3-642-33247-0_16, ISBN 978-3-642-33246-3, (MCQ). In: Question Answering for Machine ISSN 0302-9743, Springer Verlag, Berlin, Heidel- Reading Evaluation (QA4MRE) at CLEF 2013 berg, Germany. Conference and Labs of the Evaluation Forum, Va- lencia, Spain. Pinaki Bhaskar, Somnath Banerjee, and Sivaji Ban- dyopadhyay. 2012e. A Hybrid Tweet Contextual- Sibabrata Paladhi, and Sivaji Bandyopadhyay. 2008. ization System using IR and Summarization. In: the A Document Graph Based Query Focused Multi- Initiative for the Evaluation of XML Retrieval, Document Summarizer. In: the 2nd International INEX 2012 at Conference and Labs of the Evalua- Workshop on Cross Lingual Information Access tion Forum (CLEF) 2012, Pamela Forner, Jussi (CLIA), pp. 55-62. Karlgren, Christa Womser-Hacker (Eds.): CLEF Sivaji Bandyopadhyay, Amitava Das, and Pinaki 2012 Evaluation Labs and Workshop, pp. 164-175, Bhaskar. 2008. English Bengali Ad-hoc Monolin- ISBN 978-88-904810-3-1, ISSN 2038-4963, gual Information Retrieval Task Result at FIRE Rome, Italy. 2008. In: the Forum for Information Retrieval Pinaki Bhaskar, Partha Pakray, Somnath Banerjee, Evaluation (FIRE) - 2008, Kolkata, India. Samadrita Banerjee, Sivaji Bandyopadhyay, and Somnath Banerjee, Partha Pakray, Pinaki Bhaskar, Alexander Gelbukh. 2012f. Question Answering and Sivaji Bandyopadhyay, Alexander Gelbukh. System for QA4MRE@CLEF 2012. In: Question 2013. Multiple Choice Question (MCQ) Answering Answering for Machine Reading Evaluation System for Entrance Examination. In: Question (QA4MRE) at Conference and Labs of the Evalua- Answering for Machine Reading Evaluation tion Forum (CLEF) 2012, Rome, Italy. (QA4MRE) at CLEF 2013 Conference and Labs of Pinaki Bhaskar, Bidhan Chandra Pal, and Sivaji Ban- the Evaluation Forum, Valencia, Spain. dyopadhyay. 2012g. Comparative & Evaluative Udo Hahn, and Martin Romacker. 2001. The SYNDI- QA System in Tourism Domain. In: Meghanathan, KATE text generator. In: the first N., Wozniak, M.(eds.): Computational Science, International conference on Human language tech- Engineering and Information Technology: the Se- nology research, Association for Computational cond International Conference on Computational Linguistics , ACM, Morristown, NJ, USA. Science, Engineering and Informationa Technology (CCSEIT-2012), ACM International Conference Utsab Barman, Pintu Lohar, Pinaki Bhaskar, and Si- Proceeding Series. ICPS, pp. 454-460, Coimbatore, vaji Bandyopadhyay. 2012. Ad-hoc Information India. Retrieval focused on based Query Ex- pansion and Entropy Based Ranking. In: the Forum Pinaki Bhaskar, Kishorjit Nongmeikapam, and Sivaji for Information Retrieval Evaluation (FIRE) – Bandyopadhyay. 2012h. Keyphrase Extraction in 2012, Kolkata, India. Scientific Articles: A Supervised Approach. In: 24th International Conference on Computational Linguistics (Coling 2012), pp. 17-24, IIT, Bombay, Mumbai, India. Pinaki Bhaskar 2013a. A Query Focused Language Independent Multi-document Summarization. Jian, A. (Eds.), ISBN 978-3-8484-0089-8, LAMBERT Academic Publishing, Saarbrücken, Germany. Pinaki Bhaskar 2013b. Multi-document Summariza- tion using Automatic Key-phrase Extraction. In: Student Research Workshop in the Recent Ad- vances in Natural Language Processing (RANLP), Hissar, Bulgaria.

21