Welcome to the OpenHaRT 2013 Evalua8on Workshop Informaon Technology Laboratory nist.gov/itl Informaon Access Division nist.gov/itl/iad Mark Przybocki, Mul(modal Informaon Group nist.gov/itl/iad/mig August 23rd, 2013 Washington D.C. , Omni Shoreham hotel The Mul8modal Informa8on Group’s Project Areas § Speech Recogni(on § Speaker Recogni(on § Dialog Management § Human Assisted Speaker Recogni(on § Topic Detec(on and Tracking § Speaker Segmentaon § Spoken Document Retrieval § Language Recogni(on § Voice Biometrics § ANSI/NIST-ITL Standard Voice Record § Tracking (Person/Object) § Text-to-Text § Event Detec(on § Speech-to-Text § Event Recoun(ng § Speech-to-Speech § Predic(ve Video Analy(cs § Image-to-Text § Metric Development § Named En(ty Iden(ficaon § (new) Data Analy(cs § Automac Content Extrac(on 2 Defini8on: MIG’s Evalua8on Cycle Evalua'on Driven Research NIST Data NIST Researchers Performance Planning NIST Core technology Assessment development Analysis and NIST NIST Workshop 3 NIST’s MT Program’s Legacy – Past 10 Years • 27 Evaluaon Events -- tracking the state-of-the-art in performance – (4) technology types text-text speech-text speech-speech Handwri[en_Images-text – (9) languages Arabic-2, Chinese, Dari, Farsi, Hindi, Korean, Pashto, Urdu, English • 11 genres of structured and unstructured content (nwire, web, Bnews, Bconv, food, speeches, editorials, handwri(ng-2, blogs, SMS, dialogs) • 60 Evaluaon Test Sets available to MT researchers (source, references, metrics, sample system output and official results for comparison) • Over 85 research groups >400 Primary Systems Evaluated AFRL – American Univ. Cairo – Apptek – ARL – BBN – BYU – Cambridge – Chinese Acad. Sci. – 5% 3% 1% CMU – Columbia Univ. – Fujitsu Research – 11% OpenMT Google – IBM – JHU – Kansas State – KCSL – Language Weaver – Microsoh Research – Ohio TIDES State – Oxford – Qatar – Queen Mary (London) – 23% 57% TRANSTAC RWTH Aachen – SAIC - Sakhr – SRI – Stanford – Systran – UMD – USC ISI – Univ. Edinburgh – GALE Univ. of Karlsruhe – Univ. of Montreal – Xerox Research BOLT 4 .