DOCSLIB.ORG

Semantic Multimedia Modelling & Interpretation for Annotation

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Semantic Multimedia Modelling & Interpretation for Annotation Semantic Multimedia Modelling & Interpretation for Annotation Irfan Ullah Submitted in partial fulfilment of the Requirements for the degree of Doctor of Philosophy School of Engineering and Information Sciences MIDDLESEX UNIVERSITY May, 2011 © 2011 Irfan Ullah All rights reserved II Declaration I certify that the thesis entitled Semantic Multimedia Modelling and Interpretation for Annotation submitted for the degree of doctor of philosophy is the result of my own work and I have fully cited and referenced all material and results that are not original to this work. Name …………………………IRFANULLAH …. [print name] Signature …………………… …… MAY 24TH, 2011 Date………………………………. III Abstract “If we knew what it was we were doing, it would not be called research, would it?” Albert Einstein The emergence of multimedia enabled devices, particularly the incorporation of cameras in mobile phones, and the accelerated revolutions in the low cost storage devices, boosts the multimedia data production rate drastically. Witnessing such an iniquitousness of digital images and videos, the research community has been projecting the issue of its significant utilization and management. Stored in monumental multimedia corpora, digital data need to be retrieved and organized in an intelligent way, leaning on the rich semantics involved. The utilization of these image and video collections demands proficient image and video annotation and retrieval techniques. Recently, the multimedia research community is progressively veering its emphasis to the personalization of these media. The main impediment in the image and video analysis is the semantic gap, which is the discrepancy among a user’s high-level interpretation of an image and the video and the low level computational interpretation of it. Content-based image and video annotation systems are remarkably susceptible to the semantic gap due to their reliance on low-level visual features for delineating semantically rich image and video contents. However, the fact is that the visual similarity is not semantic similarity, so there is a demand to break through this dilemma through an alternative way. The semantic gap can be narrowed by counting high-level and user-generated information in the annotation. High- level descriptions of images and or videos are more proficient of capturing the semantic meaning of multimedia content, but it is not always applicable to collect this information. It is commonly agreed that the problem of high level semantic annotation of multimedia is still far from being answered. This dissertation puts forward approaches for intelligent multimedia semantic extraction for high level annotation. This dissertation intends to bridge the gap between the visual features and semantics. It proposes a framework for annotation enhancement and refinement for the object/concept annotated images and videos datasets. The entire theme is to first purify the datasets from noisy keyword and then expand the concepts lexically and commonsensical to fill the vocabulary and lexical gap to achieve IV high level semantics for the corpus. This dissertation also explored a novel approach for high level semantic (HLS) propagation through the images corpora. The HLS propagation takes the advantages of the semantic intensity (SI), which is the concept dominancy factor in the image and annotation based semantic similarity of the images. As we are aware of the fact that the image is the combination of various concepts and among the list of concepts some of them are more dominant then the other, while semantic similarity of the images are based on the SI and concept semantic similarity among the pair of images. Moreover, the HLS exploits the clustering techniques to group similar images, where a single effort of the human experts to assign high level semantic to a randomly selected image and propagate to other images through clustering. The investigation has been made on the LabelMe image and LabelMe video dataset. Experiments exhibit that the proposed approaches perform a noticeable improvement towards bridging the semantic gap and reveal that our proposed system outperforms the traditional systems. V Acknowledgement “No duty is more urgent than that of returning thanks [Schaff and Wace, 2002].” ST. AMBROSE Bishop of Milan (340 – 397) I was once told that PhD is a long journey of transformation from a ‘novice’ to ‘professional’ researcher. To succeed, an individual can never do it alone; there must be someone who is there for them, providing all the helping hands and supports. I can’t agree more and as for my case, I have a lot of people to thank. I owe my gratitude to all those people who have made this thesis possible. It is a pleasure to thank them all in my humble acknowledgment In the first place, I am heartily thankful to my supervisor Dr. Jonathan Loo (Middlesex University) for helpful discussions, guidance and support throughout my PhD studies. His scientific intuition and rigorous research attitude have enriched me with the in-depth understanding of the subject and the passion of dedicating myself to research, which I will benefit from in the long run. His gracious support in many aspects has enabled me to overcome difficulties and finish my study. I also extend my thanks to my second supervisor Dr. Martin Loomes. Further, I would like to express my special gratitude to my colleague and best friend Nida Aslam, whose dedication, love, and persistent care has diminished all the difficulties of studying and living in the unfamiliar land far away from home. I am grateful to my financial supporters, including the Kohat University of Science & Technology, Pakistan. Many thanks also go to my friends shafi Ullah khan, Tahir Naeem who accompanied me during the last three years, making my time in London most enjoyable. Finally, I pay tribute to the constant support of my family, but especially of my parents, Brother, sister and my wife & kids (Mahnoor & Lutfullah) for their untiring efforts and praying. Without them, my whole studies would have been impossible and whose sacrifice, I can never repay. This one is for you! . VI Dedication I dedicate this thesis to my late Father, (may his soul rest in peace) for their unconditional support and prayers, tireless love and motivation throughout my studies. Everything I have accomplished, I owe to him. VII List of associated Publications 1. Irfan Ullah, Nida Aslam, Jonathan Loo, RoohUllah "Adding Semantics to the reliable Object Annotated Image Databases " World Conference on Information Technology, 2010. 2. Irfan Ullah, Nida Aslam, Jonathan Loo, RoohUllah “A Framework for High Level Semantic Annotation Using Trusted Object Annotated Dataset" IEEE International Symposium on Signal Processing and Information Technology December 15-18, 2010 - Luxor – Egypt. 3. I.Khan, N. Aslam, K.K. Loo, “Semantic Annotation Gap: where to put the responsibility”, International Journal of Digital Content Technology and its Applications (JDCTA), ISSN: 1975-9339, Vol. 3, No. 1, March 2009. 4. I.Khan, N. Aslam, K.K. Loo, “Semantic Multimedia Annotation: Text Analysis”, International Journal of Digital Content Technology and its Applications (JDCTA), ISSN: 1975-9339, Vol. 3, No. 2, June 2009. 5. Irfanullah, Nida Aslam, Jonathan Loo, Roohullah. “A Framework for Image Annotation Enhancement & Refining Using Knowledge Bases”. Springerlink International Journal on Digital Libraries (Submitted Feb 2011). 6. Irfanullah, Nida Aslam, Jonathan Loo, Roohullah. “Semantic Space Enhancement and Refinement for Video using Knowledgebases”. Elsevier Journal of King Saud University – Computer and Information. (Submitted Feb 2011) 7. Irfanullah, Nida Aslam, Jonathan Loo, Roohullah. “Exploiting the Semantic Intensity for High Level Semantic Annotation of Images”. Elsevier Journal of Visual Communication and Image Representation (Submitted Feb 2011) 8. Irfanullah, Nida Aslam, Jonathan Loo, Roohullah. A survey on exploiting knowledge bases for visual media annotation. VIII Table of Contents LIST OF FIGURES ............................................................................................................................................................. XII LIST OF TABLES ............................................................................................................................................................ XVII LIST OF ACRONYMS .................................................................................................................................................... XVIII CHAPTER 01 - INTRODUCTION .................................................................................................................... 1 1.1 INTRODUCTION ................................................................................................................................... 3 1.2 MOTIVATION AND APPLICATION ........................................................................................................ 8 1.3 RESEARCH AIMS AND OBJECTIVES ..................................................................................................... 10 1.4 THE EXISTING PROBLEMS AND CHALLENGES ..................................................................................... 11 1.5 RESEARCH DIRECTIONS ..................................................................................................................... 14 1.5.1 HIGH-LEVEL SEMANTIC CONCEPTS AND LOW-LEVEL VISUAL FEATURES ............................................................... 14 1.5.2 VARIATION OF OBJECTS IN THE MULTIMEDIA
Load more

Recommended publications
  • Multimedia Data Mining a Systematic Introduction to Concepts and Theory
    Multimedia Data Mining A Systematic Introduction to Concepts and Theory © 2009 by Taylor & Francis Group, LLC C9667_FM.indd 1 10/8/08 10:06:11 AM Chapman & Hall/CRC Data Mining and Knowledge Discovery Series SERIES EDITOR Vipin Kumar University of Minnesota Department of Computer Science and Engineering Minneapolis, Minnesota, U.S.A AIMS AND SCOPE This series aims to capture new developments and applications in data mining and knowledge discovery, while summarizing the computational tools and techniques useful in data analysis. This series encourages the integration of mathematical, statistical, and computational methods and techniques through the publication of a broad range of textbooks, reference works, and hand- books. The inclusion of concrete examples and applications is highly encouraged. The scope of the series includes, but is not limited to, titles in the areas of data mining and knowledge discovery methods and applications, modeling, algorithms, theory and foundations, data and knowledge visualization, data mining systems and tools, and privacy and security issues. PUBLISHED TITLES UNDERSTANDING COMPLEX DATASETS: Data Mining with Matrix Decompositions David Skillicorn COMPUTATIONAL METHODS OF FEATURE SELECTION Huan Liu and Hiroshi Motoda CONSTRAINED CLUSTERING: Advances in Algorithms, Theory, and Applications Sugato Basu, Ian Davidson, and Kiri L. Wagsta KNOWLEDGE DISCOVERY FOR COUNTERTERRORISM AND LAW ENFORCEMENT David Skillicorn MULTIMEDIA DATA MINING: A Systematic Introduction to Concepts and Theory Zhongfei Zhang and Ruofei Zhang © 2009 by Taylor & Francis Group, LLC C9667_FM.indd 2 10/8/08 10:06:11 AM Chapman & Hall/CRC Data Mining and Knowledge Discovery Series Multimedia Data Mining A Systematic Introduction to Concepts and Theory Zhongfei Zhang Ruofei Zhang © 2009 by Taylor & Francis Group, LLC C9667_FM.indd 3 10/8/08 10:06:11 AM The cover images were provided by Yu He, who also participated in the design of the cover page.
    [Show full text]
  • How You Can Help Your Child in French Immersion
    Created by the Staff of St. Angela School French Language Arts HOW YOU CAN HELP YOUR CHILD IN FRENCH IMMERSION Here are some suggestions that will support your child in a second language program. Be positive. Just a little work and encouragement on your part can make a significant difference to your child’s attitude towards and achievement in French. • Help your child make connections in language (for example: banane - banana). • Point out French in your community, for example, signs, labels, brochures, neighbours, street names. • Provide some out-of school language and cultural experiences - . • Support your child's learning by providing the necessary tools: English/French dictionary, a dictionary of synonyms (like a thesaurus). For more information on how to support your child in French Immersion view: parentGuideFrench.pdf Yes - You Can Help Your Child in French Alberta Education Website. Homework Tool box from Ontario Created by the Staff of St. Angela School Bring French into Your Home • Use labels from can and food packages to make a collage or collect them in a scrapbook. They won’t realize they are learning vocabulary, spelling, and sorting. • Play I Spy in French. Prepare for the game by printing on cards the French names for objects in a particular room of the house. Don’t know the name of an object? Let your child know it’s OK not to know something. Make a point of finding out the word you didn't know from an older sibling, a teacher, a visual dictionary or an English/French dictionary. • Have your child label as many objects in your house as possible.
    [Show full text]
  • Beyond Flickr: Not All Image Tagging Is Created Equal
    Language-Action Tools for Cognitive Artificial Agents: Papers from the 2011 AAAI Workshop (WS-11-14) Beyond Flickr: Not All Image Tagging Is Created Equal Judith L. Klavans1, Raul Guerra1, Rebecca LaPlante1, Robert Stein2, and Edward Bachta2 1University of Maryland College Park, 2Indianapolis Museum of Art {jklavans, laplante}@umd.edu, [email protected], {rstein, ebachta}@imamuseum.org Abstract or the Art and Architecture Thesaurus (Getty 2010), and This paper reports on the linguistic analysis of a tag set of (c) clustering (Becker, Naaman, and Gravano 2010) to nearly 50,000 tags collected as part of the steve.museum characterize an image? project. The tags describe images of objects in museum col- (2) What are the optimal linguistic processes to normalize lections. We present our results on morphological, part of tags since these steps have impact on later processing? speech and semantic analysis. We demonstrate that deeper tag processing provides valuable information for organizing (3) In what ways can social tags be associated with other and categorizing social tags. This promises to improve ac- information to improve users’ access to museum objects? cess to museum objects by leveraging the characteristics of tags and the relationships between them rather than treating them as individual items. The paper shows the value of us- Description of the Tag Data ing deep computational linguistic techniques in interdisci- plinary projects on tagging over images of objects in muse- The steve project (http://www.steve.museum) is a multi- ums and libraries. We compare our data and analysis to institutional collaboration exploring applications of tagging Flickr and other image tagging projects.
    [Show full text]
  • A Visual Inter Lingua Neil Edwin Michael Leemans Worcester Polytechnic Institute
    Worcester Polytechnic Institute Digital WPI Doctoral Dissertations (All Dissertations, All Years) Electronic Theses and Dissertations 2001-04-24 VIL: A Visual Inter Lingua Neil Edwin Michael Leemans Worcester Polytechnic Institute Follow this and additional works at: https://digitalcommons.wpi.edu/etd-dissertations Repository Citation Leemans, N. E. (2001). VIL: A Visual Inter Lingua. Retrieved from https://digitalcommons.wpi.edu/etd-dissertations/154 This dissertation is brought to you for free and open access by Digital WPI. It has been accepted for inclusion in Doctoral Dissertations (All Dissertations, All Years) by an authorized administrator of Digital WPI. For more information, please contact [email protected]. VIL: A Visual Inter Lingua by Neil Edwin Michael (Paul) Leemans A Dissertation Submitted to the Faculty of the WORCESTER POLYTECHNIC INSTITUTE in partial fulfillment of the requirements for the Degree of Doctor of Philosophy in Computer Science by ____________________ April 2001 APPROVED: _____________________________________________ Dr. Lee A. Becker, Major Advisor _____________________________________________ Dr. David C. Brown, Committee Member _____________________________________________ Dr. Norman Wittels, Committee Member, Department of Civil and Environmental Engineering _____________________________________________ Dr. Stanley S. Selkow, Committee Member __________________________________________________ Dr. Micha Hofri, Head of Department VIL: A Visual Inter Lingua _____________________________________________________________________
    [Show full text]
  • Igor Burkhanov. Lexicography: a Dictionary of Basic Terminology. 1998, 285 Pp
    Resensies / Reviews 320 Igor Burkhanov. Lexicography: A Dictionary of Basic Terminology. 1998, 285 pp. ISBN 83-87288-56-X. Rzeszów: Wyższa Szkoła Pedagogiczna. Price 16 PLN. A long-awaited dictionary has at last been published, going beyond a glossary of lexicographic terms (cf. Robinson 1984) or a dictionary of a particular aspect of lexicography (cf. Cluver 1989). It is the first dictionary of basic lexicography terms. In 265 pages Burkhanov gives us a balanced view of the state of the art in dictionary form. It comprises terms typical of lexicographic work, including terms originally related to different linguistic disciplines. The selection of the data is founded, as can be inferred from the bibliogra- phy (p. 267-285), on a solid basis. It covers publications over the last twenty years, deals with various aspects of the field and includes the most recent trends from the late eighties and the nineties, e.g. Bergenholtz and Tarp (1995), Bright (1992), Bussmann (1996), Cowie (1987), Cluver (1989), Diab (1990), Fill- more and Atkins (1992), Ilson (1991), Benson, Benson and Ilson (1986), Landau (1989), Hüllen (1990), Snell-Hornby (1990). It is not difficult to find one's way in the dictionary: few symbols are used, most important of which are the ones indicating cross-references within the word-list. In this dictionary, though the arrangement of terms is alphabetical, conceptual relatedness between terms is not lost. Related terms are listed at the end of the explanatory text with the guide words "see also" and "compare". "See also" refers to superordinate and subordinate terms, e.g.
    [Show full text]
  • Promoting Fluency by Building the Sight Word Lexicon: Direct Phoneme-Grapheme Mapping and Other Techniques That Facilitate the Process of Orthographic Mapping
    Promoting Fluency by Building the Sight Word Lexicon: Direct Phoneme-Grapheme mapping and other Techniques that Facilitate the Process of Orthographic Mapping Research interest: building and developing students’ orthographic lexicons (sight word vocabularies) so students have a growing number of words they are able to read instantly and effortlessly, to support reading fluency Focus of project: Improving and remediating fluency in struggling readers by focusing on applying research about how students remember the words they read Beginning readers must develop the ability to learn to read words from memory accurately and automatically in order to read fluently Many students who struggle with reading fluency receive interventions designed to improve fluency such as repeated readings, partner reading, choral reading, and timed reading However, these interventions appear to be based on assumptions about how readers learn words that do not line up with the scientific findings about how skilled reading develops. For weak readers, repeated visual exposure to words does not substantially close the gap Two Primary Abilities of Fluent, Skilled Reading (according to David Kilpatrick) 1. Being able to identify unfamiliar words by sounding them out (word reading); achieved through phonic decoding, using visual shapes/characteristics, context, guessing 2. Being able to remember previously read words (word learning); achieved simply through recognizing words Phonics does a great job teaching students how to read words, but it does not do an adequate job of helping students make the leap to learning words so they do not have to do the work of “reading” them anymore – they just know them. Most reading fluency interventions focus on improving the first skill – identifying words through phonic decoding, visual characteristics, context, or guessing.
    [Show full text]
  • Literature Review Chinmayi Dixit & Filippa Karrfelt
    Literature Review Chinmayi Dixit & Filippa Karrfelt Our project focuses on visualizing etymological relationships between words. To this effect, the most relevant work is Etymological Wordnet: Tracing the History of Words, by De Melo, G. This work ​ ​ provides an extensive dataset to see the connections between words, across languages. Although they have a very rudimentary tool to view the results, they have not yet worked toward visualizing the dataset. ​ ​ As for the world of visual communication of language data, there has been extensive work in showing wordnet structures within the english language, for relations such as synonyms, antonyms, usage, meanings etc. However, we have not found a satisfactory visualization for the etymology of words. Some examples for these works are mentioned under visual dictionary interfaces in the reference section. One common theme that we noticed in these works, is that trees were used most extensively. The visualizations are clear to understand and well represented, when the number of nodes was small. In the case of Visualizing wordnet structure, Jaap Kamps, as the number of nodes grew large, the graph turned cluttered. Abrate et al., in WordNet Atlas: a web application for visualizing WordNet as a zoomable map, address the layout of nodes for better perception. They draw the wordnet out as a spatially arranged radial graph, with nodes arranged in concentric circles. This made large data sets more manageable and easier to navigate. One drawback, however, was that all the spatial location data had to be pre­calculated without real­time updates. We intend to work on visualizing the etymology in a cleaner and well­formatted manner.
    [Show full text]
  • Comparative Study on Currently Available Wordnets
    International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 10 (2018) pp. 8140-8145 © Research India Publications. http://www.ripublication.com Comparative Study on Currently Available WordNets Sreedhi Deleep Kumar Reshma E U PG Scholar PG Scholar Department of Computer Science and Engineering Department of Computer Science and Engineering Vidya Academy of Science and Technology Vidya Academy of Science and Technology Thrissur, India. Thrissur, India. Sunitha C Amal Ganesh Associate Professor Assistant Professor Department of Computer Science and Engineering Department of Computer Science and Engineering Vidya Academy of Science and Technology Vidya Academy of Science and Technology Thrissur, India. Thrissur, India. Abstract SinoTibetan, Tibeto-Burman and Austro-Asiatic. The major ones are the Indo-Aryan, spoken by the northern to western WordNet is an information base which is arranged part of India and Dravidian, spoken by southern part of India. hierarchically in any language. Usually, WordNet is The Eighth Schedule of the Indian Constitution lists 22 implemented using indexed file system. Good WordNets languages, which have been referred to as scheduled available in many languages. However, Malayalam is not languages and given recognition, status and official having an efficient WordNet. WordNet differs from the encouragement. dictionaries in their organization. WordNet does not give pronunciation, derivation morphology, etymology, usage notes, A Dictionary can be called as are source dealing with the or pictorial illustrations. WordNet depicts the semantic relation individual words of a language along with its orthography, between word senses more transparently and elegantly. In this pronunciation, usage, synonyms, derivation, history, work, a general comparison of currently browsable WordNets etymology, etc.
    [Show full text]
  • Japanese-English Bilingual Visual Dictionary Ebook, Epub
    JAPANESE-ENGLISH BILINGUAL VISUAL DICTIONARY PDF, EPUB, EBOOK DK | 360 pages | 01 Feb 2018 | Dorling Kindersley Ltd | 9780241317556 | English | London, United Kingdom Japanese-English Bilingual Visual Dictionary PDF Book To find out more, including how to control cookies, see here: Cookie Policy. Picture dictionaries. Whether you're travelling for business or leisure, buying food or train tickets, discussing work or famous sights on the tourist trail, you'll gain confidence in your new language skills with this bilingual visual dictionary by your side. Charles Darwin. Using full color photographs and artworks to display and label all the elements of everyday life from the home and office to sports, music, and nature the Japanese English Bilingual Visual Dictionary has over 1, photographs and illustrations, more than 6, annotated terms. Admin Admin Admin, collapsed. Become a Member Start earning points for buying books! This handy travel-sized guide also includes a detailed index for instant reference. You are commenting using your Google account. Email required Address never made public. Wanda C. We have a reputation for innovation in design for both print and digital products. Japanese-English Bilingual Visual Dictionary. Christianbook Prefilled Communion Cups, Box of Margot Adler. The Japanese and English Bilingual Visual Dictionary comes in a travel-sized and with a pronunciation guide so you will never be stuck for the right word again. The audio app, available for Apple from the App Store and Android from Google Play , enables you to hear words and phrases spoken by native Japanese speakers. Periplus Pocket Japanese Dictionary. Please enter your name, your email and your question regarding the product in the fields below, and we'll answer you in the next hours.
    [Show full text]
  • Italian English Bilingual Visual Dictionary (DK Visual Dictionaries) by DK
    Italian English Bilingual Visual Dictionary (DK Visual Dictionaries) by DK Ebook Italian English Bilingual Visual Dictionary (DK Visual Dictionaries) currently available for review only, if you need complete ebook Italian English Bilingual Visual Dictionary (DK Visual Dictionaries) please fill out registration form to access in our databases Download here >> Series:::: DK Visual Dictionaries+++Paperback:::: 360 pages+++Publisher:::: DK; Bilingual, Revised, Updated edition (April 18, 2017)+++Language:::: English+++ISBN-10:::: 1465459308+++ISBN-13:::: 978-1465459305+++Product Dimensions::::5.5 x 1 x 6.5 inches++++++ ISBN10 1465459308 ISBN13 978-1465459305 Download here >> Description: Now comes with a free companion audio app that allows readers to scan the pages to hear words spoken in both Italian and English.Newly revised and updated, the Italian–English Bilingual Visual Dictionary is a quick and intuitive way to learn and recall everyday words in Italian.Introducing a range of useful current vocabulary in thematic order, this dictionary uses full-color photographs and artworks to display and label all the elements of everyday life—from the home and office to sport, music, nature, and the countries of the world—with panel features on key nouns, verbs, and useful phrases.The Italian–English Bilingual Visual Dictionary features:• A quick and intuitive way to learn and remember thousands of words.• A complete range of illustrated objects and scenes from everyday life.• Fast and effective learning for any situation, from home and office to shopping and dining out.• Detailed index for instant reference.• Handy size ideal for travel.The illustrations provide a quick and intuitive route to learning a language, defining the words visually so it is easier to remember them and creating a colorful and stimulating learning resource for the foreign-language and EFL/ESL student.
    [Show full text]
  • Logic Based Methods for Terminological Assessment
    Logic Based Methods for Terminological Assessment Benoît Robichaud Observatoire de linguistique Sens-Texte (OLST) Université de Montréal [email protected] Abstract We present a new version of a Graphical User Interface (GUI) called DiCoInfo Visuel, mainly based on a graph visualization device and used for exploring and assessing lexical data found in DiCoInfo, a specialized e-dictionary of computing and the Internet. This new GUI version takes advantage of the fundamental nature of the lexical network encoded in the dictionary: it uses logic based methods from logic programming to explore relations between entries and find pieces of relevant information that may be not accessible by direct searches. The result is a more realistic and useful data coverage shown to end users. Keywords: electronic dictionary, Graphical User Interface, data visualization, logic programming, assessment tool 1. Introduction relationships that remained hidden in the preceding This paper describes the goals, architecture and usability version; second, search recursion is used to compute of a new version of a Graphical User Interface (GUI) transitive closures on selected (or all) subsets of lexical used to browse and assess encoded data through a subset relations allowing the implementation of other search of selected lexical relations found in DiCoInfo strategies and layout improvements. (Dictionnaire fondamental de l’informatique et de l’Internet), an online multilingual specialized dictionary 2. Motivation and related work of computing and the Internet. The GUI allows searches This GUI project started with the idea that it was in the dictionary entry fields and presents results as possible to improve the visual and communicative value (directed) graphs.
    [Show full text]
  • Comparing Phonemes and Visemes with DNN-Based Lipreading
    THANGTHAI, BEAR, HARVEY: PHONEMES AND VISEMES WITH DNNS 1 Comparing phonemes and visemes with DNN-based lipreading Kwanchiva Thangthai1 1 School of Computing Sciences [email protected] University of East Anglia Helen L Bear2 Norwich, UK [email protected] 2 Computer Science & Informatics Richard Harvey1 University of East London [email protected] London, UK. Abstract There is debate if phoneme or viseme units are the most effective for a lipreading system. Some studies use phoneme units even though phonemes describe unique short sounds; other studies tried to improve lipreading accuracy by focusing on visemes with varying results. We compare the performance of a lipreading system by modeling visual speech using either 13 viseme or 38 phoneme units. We report the accuracy of our system at both word and unit levels. The evaluation task is large vocabulary continuous speech using the TCD-TIMIT corpus. We complete our visual speech modeling via hybrid DNN-HMMs and our visual speech decoder is a Weighted Finite-State Transducer (WFST). We use DCT and Eigenlips as a representation of mouth ROI image. The phoneme lipreading system word accuracy outperforms the viseme based system word accuracy. However, the phoneme system achieved lower accuracy at the unit level which shows the importance of the dictionary for decoding classification outputs into words. arXiv:1805.02924v1 [cs.CV] 8 May 2018 1 Introduction As lipreading transitions from GMM/HMM-based technology to systems based on Deep Neural Networks (DNNs) there is merit in re-examining the old assumption that phoneme- based recognition outperforms recognition with viseme-based systems.
    [Show full text]