An Attempt to Measure the Quality of Questions in Question Time of the Australian Federal Parliament
Andrew Turpin University of Melbourne [email protected]
ABSTRACT Question Time in the House of Representatives is This paper uses standard information retrieval techniques to often criticised for declining parliamentary stan- measure the quality of information exchange during Ques- dards and accountability. Oppositions are in- tion Time in the Australian Federal Parliament’s House of clined to use partisan attacks disguised as ques- Representatives from 1998 to 2012. A search engine is used tions to embarrass the government to which Min- to index all answers to questions, and then runs each ques- isters respond with lengthy answers of marginal tion as a query, recording the rank of the actual answer in relevance. Ministers often use Question Time to the returned list of documents. Using this rank as a mea- attack the opposition with pre-prepared state- sure of quality, Question Time has deteriorated over the last ments in response to “Dorothy Dix” questions decade. The main deterioration has been in information ex- from their own side. Much of the theatre of Ques- change in “Dorothy Dixer” questions. The corpus used for tion Time is characterised by disorder contrived this study is available from the author’s web page for further to make the evening news [9]. investigations. In this paper we use an off-the-shelf, open source informa- tion retrieval (IR) system to quantify whether questions and Categories and Subject Descriptors answers in Question Time contain less information and rel- evance in recent years than in past years. IR systems index H.3 [Information Storage and Retrieval]: Miscellaneous a collection of documents, and then for any provided query, rank the documents in the collection against the query, with General Terms the top ranked document being deemed the most relevant to the query. In this study, all answers to all questions from Application Question Time in the period 1998 through to 2012 were in- dexed as a single document collection, then each question Keywords was run against the collection as a query. In theory, the an- swer provided by a parliamentarian to a question should be Information retrieval, Question Time, Parliament, Search the top ranked document returned by the system; assuming, Engine of course, that the question was answerable, and that they actually provided the answer. The hypothesis tested by this 1. INTRODUCTION paper, therefore, is that the rank of answer documents to questions posed in 2012 are lower than the rank of answer Question Time in the House of Representatives in the Aus- documents in 1998. That is, questions and answers in 1998 tralian Federal Parliament begins at 2pm on every sitting contained more content that was decipherable by a search day, and is a period of about one hour where members of engine than 2012. the parliament can ask Ministers questions and receive an Hansard is a written daily record of everything that occurs answer typically limited to several minutes in length. While in the parliament. The next section discusses the creation of a seemingly essential part of a functioning democracy, Ques- the Question Time corpus from downloaded Hansard XML tion Time has come under attack in recent days [10], and files: a non-trivial task, as it turned out. Then we present after the last election [1], for lacking content and relevance. some results to address the hypothesis, and the final sec- Mark Rodrigues’ publication on parliamentary reform from tion of the paper discusses the results and limitations of this 2010 sums it up as follows. study. 2. CORPUS CREATION Permission to make digital or hard copies of all or part of this work for All of the Hansard from 1998 to the current day can be personal or classroom use is granted without fee provided that copies are downloaded from the Australian Parliament House Hansard not made or distributed for profit or commercial advantage and that copies Web site [4] in XML format. Unfortunately, from a compu- bear this notice and the full citation on the first page. To copy otherwise, to tational point of view, the XML format changed mid-2011, republish, to post on servers or to redistribute to lists, requires prior specific and so different parsers are required to extract information permission and/or a fee. ADCS ’12 December 05 - 06 2012, Dunedin, New Zealand for Hansards after March 2011 and before. This section Copyright 2012 ACM 978-1-4503-1411-4/12/12 ...$15.00. describes how we extracted questions and answers from the XML. The final collection is available for download in TREC files in 2012 and 1999. If those years are representative, then format from the author’s home page. there are most likely 5 to 10 errors in the other years that The extraction begins with an XSL script to extract the remained uncorrected, and so may have filtered through to “Questions Without Notice” 5. LP NATS NP Nats NaysWA NPActing CLP 6. UNKNOWN N/A Mr Laming: Thus if the party of the questioner and responder fell into one of these categories, the question was deemed a Dorothy Dixer. Superclinics? The final corpus contained 16310 question-answer pairs as outlined in Table 1 from 53 periods of parliament covering tion as query and record the rank of the answer. For this purpose, we used the Zettair search engine (version 0.9.3) [12] with default Okapi BM25 [7, 8] parameters, stemming and The DEPUTY SPEAKER: stop list. The index created over all answers contained 16,310 documents, 54,971 distinct index terms, and 6,929,465 The member for Wannon has got one last chance. terms. Figure 1 shows the median and inter-quartile ranges (IQR)
Answers Mean Len. 168 157 181 198 190 178 182 138 144 181 217 245 220 192 154 Max Len. 503 718 563 763 771 635 533 375 442 498 777 839 620 326 306
Questions Mean Len. 32 30 31 31 31 32 30 29 31 26 21 25 25 25 25 Max Len. 191 114 157 91 126 146 141 132 159 133 130 137 158 81 100
● ● 180 14 160 ● ●
● 140 12 ● ● ● ● ●
120 ● ●
10 ● ● ● 100
● IQR rank ●
Median rank ● ● ● ● 8 80 ● ●
● 60 ● 6
● ● ● 40 ● ●
1998 2002 2006 2010 1998 2002 2006 2010 Year Year
Figure 1: Median and inter-quartile ranges (IQR) of the rank of the answer document for each question. compared relatively within a corpus. Again, lines shown are IR engine only gets about 20% of matching answers in rank simple linear regression lines. Surprisingly, Dorothy Dixers position one. This seems to match public opinion that ques- show a significant increase in rank (p = 0.002) and decrease tions and answers in Question Time have less informative in score (p < 0.001) over time. Questions that were not content than general parliamentary debate. Dorothy Dixers did not show a significant change in rank (p = 0.454), nor score (p = 0.030). This provides some evi- 4. NEAR DUPLICATES dence for Rodrigues’ claim that Dorothy Dixers are increas- It is conceivable that some answers are similar to another, ingly used as a platform for making party political state- and so it may be unfair to expect an IR system to find ex- ments that have little relevance to the question. actly the right answer to a specific question from a pool of documents that are very similar. In fact, 59 answer docu- Questions in Writing ments are one word, “No.”, and 8 are “Yes”. Browsing doc- As an alternate experiment, the XSL scripts were modified uments of 15 words or fewer reveals that very few of them to extract“QUESTIONS IN WRITING”from Hansard. The would contain terms that would match a specific question post processing scripts were also altered as the Questions in using any TFxIDF style IR metric: many are simply wordy Writing had no interjections, and a slightly different intro- ways of saying “no”. For example, “The answer to the hon- ductory sentence. 7032 question-answer pairs were extracted ourable gentleman’s question is no” or “In precise answer to from the corpus, covering years 2004 onwards as described in the tail end of the question: no”. There are 251 documents the first row of Table 2. As can be seen in the final three rows with 15 words or less. of the table, the IR engine ranks the answer to a question Taking this one step further, if an answer is run as a query in position one generally over 50% of the time: far more against the collection of answers, where will itself rank? If success than on questions from Question Time, where the the answer is unambiguous according to the IR engine, then ● ● ● DD ● non−DD 45 50 ● ●
● ● ● 40
40 ● ●
● ● ● ● 35
30 ● ● ● ● ● ● ●
Median rank ● ● ● 30 ● ● ● ●
20 ● ● ● ● ● Median similarity score ● ● ● 25 ● ● ●
10 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 20
1998 2002 2006 2010 1998 2002 2006 2010 Year Year
Figure 2: Median and inter-quartile ranges (IQR) of the rank of the answer document for each question separated by Dorothy Dixer (filled) and non-Dorothy Dixer (open).
Table 2: Statistics on the Questions in Writing corpus. The last three rows show a summary of ranks of the answer documents as ranked by Okapi BM25 when the question is posed as a query. Year 2004 2005 2006 2007 2008 2009 2010 2011 2012 Number 5 1841 1560 1205 303 588 279 496 330 Med. Rank 2 1 2 2 1 1 1 1 1 IQR Rank 1 12 16 10 3 4 3 5 9 % at Rank 1 40% 50% 47% 44% 58% 55% 64% 58% 53% it should appear at rank position one. There are 91 docu- ter may not be responsible, as Howard was PM both before ments where this is not the case. These documents have an and after November 2004. Perhaps the quality of Question average length of about 8 words, with the longest being 22 Time as an information exchange is driven by the Opposi- words. tion Leader, who is the chief questioner, or the Speaker of A similar exercise can be undertaken for questions. Build- the House, who chairs the debate and monitors content. ing a document collection of just the questions as documents, Figure 5 shows the ranks of returned answers grouped by and then running each question as a query, 44 do not come the two people over the lifetime of the corpus. It is not up with themselves as the first ranked document, 39 of which that illuminating: no one obviously stands out as a culprit. are Dorothy Dixers. The majority of these questions are Kim Beazley is the only person in the corpus to be opposi- of the form: “Would the minister please update the House tion leader twice, and there is a significant increase in ranks about...”. Perhaps some more sophisticated phrase stopping (Wilcoxon p < 0.001) between his two terms, perhaps indi- technique might distill these questions to their essence, but cating that he alone did not control the content of Question this was not tried in this work. Time in that period. If we remove all question-answer pairs where one or the Perhaps the only observation that can be drawn from this other does not return itself as the top ranked document, part of the analysis is that the 6 sessions from February or the answer is 15 words or less, this excludes 386 pairs 2002 to December 2003 (Prime Minister Howard, Opposition of questions and answers. Not surprisingly, this made no Leader Crean, Speaker Andrew) had low ranks relative to difference to the trend in median ranks or IQRs shown in other periods where personnel remained unchanged. Figure 1. 6. ON MESSAGE 5. PERSONNEL It is currently common practice for parliamentarians from Figure 3 shows the median and IQR of ranks for each sit- the major parties to be issued with “talking points” for a ting period (typically about twenty days). One could conjec- day, and there is a strong emphasis to “stay on message”. ture that the 39th and 40th parliaments (prior to November Perhaps, therefore, the answers given to questions, particu- 2004) contained Question Times that were more informative larly Dorothy Dixers, are very similar, making it difficult for than that of the 41st, 42nd and current 43rd parliament post an IR engine to pick the correct answer to any given ques- November 2004. One could also argue that the Prime Minis- tion. To explore this possibility, we compared the scores of ● ● ● Howard Howard Howard Rudd Gillard
Howard Howard Howard Rudd Gillard 250
● 20 ● ● ● ● 200
● ● ● 15 ● ● ●
● ● ●● 150 ● ● ●● ●
IQR rank ● ● ● ●
Median rank ● ● ● ●● ● ● ●● ●
10 ● ● ● ● ● ●● ● ● ● ● ● ● ● ●
● 100 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● 5 ● 50 ● ● ● ● ● ●●● ● ● ● Nov Feb Nov Feb Sep Aug Nov Feb Nov Feb Sep Aug 1998 2002 2004 2008 2010 2012 1998 2002 2004 2008 2010 2012
Figure 3: Median and inter-quartile ranges (IQR) of the rank of the answer document for each question with one circle for each sitting period. Dotted lines show a change in parliament, with the names of the Prime Minister at the top of each parliament. The diamond gives the median value over the whole parliament.
period and median ∆a (Pearson, p = 0.4). But, there is a ●
0.66 linear trend for median ∆a decreasing from about 66% in 1998 to 61% in 2012 (p = 0.007). Figure 4 shows the data. ● Splitting the data into Dorothy Dixers and non Dorothy
0.65 Dixers reveals little. Both follow a similar pattern as the aggregated data, decreasing from bout 66% to 61%. ●
0.64 ● ● ● 7. DISCUSSION
● This paper attempts to objectively measure the quality of 0.63 ● Question Time in the House of Representatives of the Aus- tralian Federal Parliament using state-of-the-art document ● ● ● ranking schemes that are the underlying technology of mod- 0.62 Relative diff. top 2 scores diff. Relative ● ern search engines. All answers given in Question Time in ● the period 1998 through to August 2012 are collected and ●
0.61 ● indexed by the IR system, then each question is posed as a query to the system and the rank of its known answer recorded. The IR engine has more trouble finding answers 1998 2002 2006 2010 to questions in recent years than in the past, leading to the Year conclusion that Question Time has reduced in quality over the last decade. Figure 4: Median relative difference in the Okapi Throughout the experiments the Okapi BM25 metric [7, 8] scores of the top two answer documents returned with default settings was used as the ranking metric. We also when each answer is run as a query. generated Figure 1 using a Language Model scheme [6] with Dirichlet smoothing [13] using µ = 2000 and µ = 500, but there was no appreciable difference from the Okapi results, the top two ranked documents when using each answer as a hence they are not reported in this paper. Perhaps there are query. other similarity metrics that are more suited to this retrieval For each answer document a, define ∆a to be the differ- task. Going beyond the standard approaches to include the ence between the top two scores of documents returned when meta-data identifying speakers, parties, and so on, or even a is issued as query, divided by the top score. Thus if ∆a is exploiting the audio recordings of question time to determine high, the second document is scored as very different from speech patterns would be interesting areas for future study. the first, while a low ∆a indicates that the top two doc- While the length of questions in words has decreased in uments are similar, according to the IR engine’s similarity recent years (see Table 1), the marked increase in median measure. There is no correlation, between the median rank rank of answers occurs for 2003 where question length was of answers as returned by the IR engine in a parliamentary still on a par with previous years. It isn’t until 2007 that a” hnrdcn h otn fQeto ieprl to purely Time Question of content every- the the into reducing process rep- Mary then political it day”, to the worse, of subscribe or translation you the better If resents “For that Time. [3] Question observation of Crawford’s aim work. the as future sees in avenue this explore will instruction We and planning annotators. careful to requires set anno- data Time, compli- Question this in the tating exchange of information all the of Given of cations judgements required. human are data, pairs Time question-answer Question of quality corpus. of this sure This using research component. future IR for the area on fertile not a and like recogni- process, seems speech Re- the the Word of on Spoken side predominately into tion research focused that has varia- note [5] this trieval aside, of an none As captures documents. here tion. written used in ranking found TFxIDF usually The not pleasantries it social that include ugliness) are so (or word, here their spoken used the in “documents” the of devices transcripts Also any Vaudeville answers. and and and rhetorical interjections, questions of to metaphor, manner retorts doublespeak, other question herrings, include Time red Question Politicians half-truths, in myr- present the answers. are incorporate and that however, nuances not, of does iad to the It queries currently matching [2]. is efficiently documents approach for occur frequently technology This that available occur (IDF). words best not documents do query be- and other if (TF) score in query document similarity a a in the TFxIDF and frequently increases a document approaches scheme a using a IR tween documents Such the ranks of answersscheme. and and All bags-of-words, questions treat as stopping, limitations. stemming, its use tested has Time can tion answers or questions results. in there- the words unlikely, for Leader of is account number Opposition It the and that shorter. (left) fore, noticeably Speaker become by questions grouped question each for documents answer of (right). Ranks 5: Figure n ugmn fase ult eed nwa one what on depends quality answer of judgement Any mea- a as rankings TFxIDF of use the validate to order In Ques- of quality measure to system IR an using Obviously
Rank
0 50 100 150 200 250 300
Sinclair Andrew
Speaker
Hawker
Jenkins Slipper
o epu icsin.Takyut h nnmu re- by supported anonymous is the research FT0991326. to Grant This ARC you comments. useful Thank for viewers discussions. helpful for ACKNOWLEDGMENTS 9. with compared when writing. poor in question-answer consistently questions other remained in has information the pairs the Surprisingly, over while end decreased period, the 2012. has this Dixers since to Dorothy deterioration in through to contained continues sharp 1998 information that a from decreased 2003 been has of has system, There IR an 2012. by measured as CONCLUSION 8. documents. answer of primary rank required. the median be is could investigation increased Further this the and on, of goes similar cause time more content as becoming information other are each the Dixers welcome. to Dorothy that be to show answers may 6 the of information Section in of results repetition The the of duction an- manual any data. to the challenge of significant notation retrieval a information also cur- and and to systems, challenge analysis major language a automatic presents rent which of information text obstruct a the not likely sharing, does most theatre is then There where theatre, apply. ground, directly middle than life paper rather this politician’s in sharing, Australian results the information an in one of If used hour was an exercise. futile that a rather is would problem matching information an Rank
hnst i adi,Aiti oa n akScholer Falk and Moffat Alistair Baldwin, Tim to Thanks Time, Question in exchanged information of quality The re- Time, Question of content theatrical the of Regardless 0 50 100 150 200 250 300
Beazley Crean
Opposition Leader Latham
Beazley
Howard
Nelson
Turnball Abbott APPENDIX [5] M. Larson and G. J. F. Jones. Spoken content retrieval: A survey of techniques and technologies. A. TECHNICAL DETAILS OF CORPUS Foundations and Trends in Information Retrieval, CONSTRUCTION 5(4-5):235–422, 2011. Figure 6 shows the final XSL used. Prior to March 2011, [6] J. M. Ponte and W. B. Croft. A language modelling interjections were tagged separately as
7. Extended ASCII characters were mapped to standard ASCII characters in both questions and answers. In particular: the Euro and Pound symbols were mapped to EURO and GBP respectively; various length hyphens were all mapped to a simple minus sign; and various quotation marks were mapped to simple single quotes. Double quotes " were also substituted with a single ’ to prevent phrase processing by the IR engine.
B. REFERENCES [1] AAP. Squabble breaks out over parliamentary reform. Sydney Morning Herald, Sept. 2010. [2] T. G. Armstrong, A. Moffat, W. Webber, and J. Zobel. Improvements that don’t add up: ad-hoc retrieval results since 1998. In Proceedings of the 18th ACM conference on Information and knowledge management, CIKM ’09, pages 601–610, New York, NY, USA, 2009. ACM. [3] M. Crawford. Question Time: don’t change the contest we want to watch. http://theconversation.edu.au, Sept. 2012. [4] http://www.aph.gov.au/Parliamentary_Business/ Hansard.
Figure 6: The XSL used for extracting the first question in each subdebate from Hansard XML published after March 2011. Answers are extracted similarly, replacing “question” with “answer” in line 3.
Figure 7: The XSL used for extracting the first question in each subdebate from Hansard XML published before March 2011. Answers are extracted similarly, replacing “question” with “answer” in lines 2 and 3.
s/Opposition members interjecting//g s/Government members interjecting//g s/Honourable members interjecting//g s/Opposition member interjecting//g s/Government member interjecting//g s/Honourable member interjecting//g s/A government member interjecting//g s/An opposition member interjecting//g s/Mr [a-zA-Z’"’"’]* interjecting//g s/Mrs [a-zA-Z’"’"’]* interjecting//g s/Ms [a-zA-Z’"’"’]* interjecting//g s/Dr [a-zA-Z’"’"’]* interjecting//g s/Mr [a-zA-Z]* [a-zA-Z’"’"’]* interjecting//g s/Mrs [a-zA-Z]* [a-zA-Z’"’"’]* interjecting//g s/Ms [a-zA-Z]* [a-zA-Z’"’"’]* interjecting//g s/Dr [a-zA-Z]* [a-zA-Z’"’"’]* interjecting//g s/The member for [A-Za-z]* is interjecting//g s/The member for [A-Za-z]* [A-Za-z]* is interjecting//g s/The member for [A-Za-z]*-[A-Za-z]*is interjecting//g
Figure 8: Sed script used to remove interjections that were not tagged in the XML.