DOCSLIB.ORG

A Visual Turing Test for Computer Vision Systems

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A Visual Turing Test for Computer Vision Systems i i “Manuscript-DEC152014” — 2014/12/16 — 9:40 — page 1 — #1 i i A visual Turing test for computer vision systems Donald Geman ∗, Stuart Geman y, Neil Hallonquist ∗ and Laurent Younes ∗ ∗Johns Hopkins University, and yBrown University Submitted to Proceedings of the National Academy of Sciences of the United States of America Today, computer vision systems are tested by their accuracy in de- of challenges and competitions (see [4, 5]) suggest that progress has tecting and localizing instances of objects. As an alternative, and been spurred by major advances in designing more computationally motivated by the ability of humans to provide far richer descriptions, efficient and invariant image representations [6, 7, 8]; in stochastic even tell a story about an image, we construct a "visual Turing test": and hierarchical modeling [9, 10, 11, 12]; in discovering latent struc- an operator-assisted device that produces a stochastic sequence of ture by training multi-layer networks with large amounts of unsuper- binary questions from a given test image. The query engine pro- poses a question; the operator either provides the correct answer vised data [13]; and in parts-based statistical learning and modeling or rejects the question as ambiguous; the engine proposes the next techniques [14, 15, 16], especially combining discriminative part de- question (\just-in-time truthing"). The test is then administered tectors with simple models of arrangements of parts [17]. Quite re- to the computer-vision system, one question at a time. After the cently, sharp improvements in detecting objects and related tasks have system's answer is recorded, the system is provided the correct an- been made by training convolutional neural networks with very large swer and the next question. Parsing is trivial and deterministic; the amounts of annotated data [18, 19, 20, 21, 22]. system being tested requires no natural language processing. The More generally, however, machines lag very far behind humans in query engine employs statistical constraints, learned from a training “understanding images” in the sense of generating rich semantic an- set, to produce questions with essentially unpredictable answers| notation. For example, systems that attempt to deal with occlusion, the answer to a question, given the history of questions and their correct answers, is nearly equally likely to be positive or negative. context and unanticipated arrangements, all of which are easily han- In this sense, the test is only about vision. The system is designed dled by people, typically encounter problems. Consequently, there is to produce streams of questions that follow natural story lines, from no point in designing a “competition” between computer vision and the instantiation of a unique object, through an exploration of its human vision: interpreting real scenes (such as the ones in Figure 1) properties, and onto its relationships with other uniquely instanti- is virtually “trivial” (at least effortless and nearly instantaneous) for ated objects. people whereas building a “description machine” that annotates raw image data remains a fundamental challenge. scene interpretation j computer vision j Turing test j binary questions j un- We seek a quantitative measure of how well a computer vision sys- predictable answers j sparse learning tem can interpret ordinary images of natural scenes. Whereas we focus on urban street scenes, our implementation could easily be ex- Going back at least to the mid-twentieth century there has been an tended to other image populations and the basic logic and motivations active debate about the state of progress in artificial intelligence and remain the same. The “score” of our test is based on the responses of how to measure it. Alan Turing [1] proposed that the ultimate test of a system under evaluation to a series of binary questions about the whether or not a machine could “think,” or think at least as well as a existence of people and objects, their activities and attributes, and person, was for a human judge to be unable to tell which was which relationships among them, all relative to an image. We have chosen based on natural language conversations in an appropriately cloaked image-based rather than scene-based queries (see Scenes vs. images). scenario. In a much-discussed variation (sometimes called the “stan- Suppose an image sub-population I has been specified (“urban dard interpretation”), the objective is to measure how well a com- street scenes” in Figure 1), together with a “vocabulary” and a cor- puter can imitate a human [2] in some circumscribed task normally responding set of binary questions (see Vocabulary and Questions). associated with intelligent behavior, although the practical utility of Our prototype “visual Turing test” (VTT) is illustrated in Figure 2. “imitation” as a criterion for performance has also been questioned Questions are posed sequentially to the computer vision system using [3]. In fact, the overwhelming focus of the modern AI community has been to assess machine performance more directly by dedicated tests for specific tasks rather than debating about general “thinking” or Turing-like competitions between people and machines. Significance In this paper we implement a new, query-based test for computer vision, one of the most vibrant areas of modern AI research. Through- In computer vision, as in other fields of AI, the methods of eval- out this paper we use “computer vision” more or less synonymously uation largely define the scientific effort. Most current evalua- tions measure detection accuracy, emphasizing the classification with semantic image interpretation - “images to words.” But of course of regions according to objects from a pre-defined library. But computer vision encompasses a great many other activities; it in- detection is not the same as understanding. We present here a cludes the theory and practice of image formation (“sensors to im- different evaluation system, in which a query engine prepares a ages”); image processing (“images to images”); mathematical rep- written test (\visual Turing test") that uses binary questions to resentations; video processing; metric scene reconstruction; and so probe a system's ability to identify attributes and relationships in addition to recognizing objects. forth. In fact, it may not be possible to interpret scenes at a semantic level without taking at least some of these areas into account, espe- Reserved for Publication Footnotes cially the geometric relationship between an image and the underly- ing 3D scene. But our focus is how to evaluate a system, not how to build one. Besides successful commercial and industrial applications, such as face detectors in digital cameras and flaw detection in manufactur- ing, there has also been considerable progress in more generic tasks, such as detecting and localizing instances from multiple generic ob- ject classes in ordinary indoor and outdoor scenes, in “fine-grained” classification such as identifying plant and animal species, and in rec- ognizing attributes of objects and activities of people. The results www.pnas.org/cgi/doi/10.1073/pnas.0709640104 PNAS Issue Date Volume Issue Number 1{7 i i i i i i “Manuscript-DEC152014” — 2014/12/16 — 9:40 — page 2 — #2 i i Fig. 1. Urban street scenes. First row: Athens, Baltimore, Busan, Delhi. Second row: Hong Kong, Miami, Rome, Shanghai. a “query language” which is defined in terms of an allowable set of human annotators? The reason, and the justification for the VTT, is predicates. The interpretation of the questions is unambiguous and that the current method does not scale with respect to the richness of does not require any natural language processing. The core of the the representation. Even for the sub-tasks in the competitions men- VTT is an automatic “query generator” which is learned from an- tioned earlier, the evaluation of performance, i.e., comparing the out- notated images and produces a sequence of binary questions for any put of the system (e.g., estimated bounding boxes) to the ground-truth given “test” image I0 2 I whose answers are “unpredictable” (see is not always straightforward and the quality of matches must be as- Statistical Formulation). In loose terms, this means that hearing the sessed [28]. Moreover, annotating every image submitted for testing first k − 1 questions and their true answers for I0 without actually at massive levels of detail is not feasible. Hence, objectively scoring seeing I0 provides no information about the likely answer to the next the veracity of annotations is not straightforward. As in school, an- question. In order to prepare for the test, designers of the vision sys- swering specific questions is usually more objective and efficient in tems would be provided with the database used to train the query measuring “understanding.” Finally, some selection procedure seems generator as well as the full vocabulary and set of possible questions, unavoidable; indeed, the number of possible binary questions which and would have to provide an interface for answering questions. One are both probing and meaningful is virtually infinite. However, select- simple measure of performance is the average number of correct re- ing a subset of questions (i.e., preparing a test) is not straightforward. sponses over multiple runs with different test images. We would argue that the only way to ask very detailed questions with- Current Evaluation Practice out having their answers be almost certainly “no” is sequential and adaptive querying—questions which build on each other to uncover Numerous datasets have been created to benchmark performance, semantic structure. In summary, the VTT is one way to “scale up” each designed to assess some vision task (e.g., object detection) on evaluation. some image domain (e.g., street-scenes). Systems are evaluated by comparing their output on these data to “ground-truth” provided by Proposed Test: Overview humans. One well-studied task is classifying an entire image by a Images of scenes.
Load more

Recommended publications
  • Fake It Until You Make It Exploring If Synthetic GAN-Generated Images Can Improve the Performance of a Deep Learning Skin Lesion Classifier
    Fake It Until You Make It Exploring if synthetic GAN-generated images can improve the performance of a deep learning skin lesion classifier Nicolas Carmont Zaragoza [email protected] 4th Year Project Report Artificial Intelligence and Computer Science School of Informatics University of Edinburgh 2020 1 Abstract In the U.S alone more people are diagnosed with skin cancer each year than all other types of cancers combined [1]. For those that have melanoma, the average 5-year survival rate is 98.4% in early stages and drops to 22.5% in late stages [2]. Skin lesion classification using machine learning has become especially popular recently because of its ability to match the accuracy of professional dermatologists [3]. Increasing the accuracy of these classifiers is an ongoing challenge to save lives. However, many skin classification research papers assert that there is not enough images in skin lesion datasets to improve the accuracy further [4] [5]. Over the past few years, Generative Adversarial Neural Networks (GANs) have earned a reputation for their ability to generate highly realistic synthetic images from a dataset. This project explores the effectiveness of GANs as a form of data augmentation to generate realistic skin lesion images. Furthermore it explores to what extent these GAN-generated lesions can help improve the performance of a skin lesion classifier. The findings suggest that GAN augmentation does not provide a statistically significant increase in classifier performance. However, the generated synthetic images were realistic enough to lead professional dermatologists to believe they were real 41% of the time during a Visual Turing Test.
    [Show full text]
  • Boundary Detection by Constrained Optimization ‘ DONALD GEMAN, MEMBER, IEEE, STUART GEMAN, MEMBER, IEEE, CHRISTINE GRAFFIGNE, and PING DONG, MEMBER, IEEE
    IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. VOL. 12. NO. 7, JULY 1990 609 Boundary Detection by Constrained Optimization ‘ DONALD GEMAN, MEMBER, IEEE, STUART GEMAN, MEMBER, IEEE, CHRISTINE GRAFFIGNE, AND PING DONG, MEMBER, IEEE Abstract-We use a statistical framework for finding boundaries and tations related to image segmentation: partition and for partitioning scenes into homogeneous regions. The model is a joint boundary labels. These, in tum, are special cases of a probability distribution for the array of pixel gray levels and an array of “labels.” In boundary finding, the labels are binary, zero, or one, general “label model” in the form of an “energy func- representing the absence or presence of boundary elements. In parti- tional’’ involving two components, one of which ex- tioning, the label values are generic: two labels are the same when the presses the interactions between the labels and the (inten- corresponding scene locations are considered to belong to the same re- sity) data, and the other encodes constraints derived from gion. The distribution incorporates a measure of disparity between cer- general information or expectations about label patterns. tain spatial features of pairs of blocks of pixel gray levels, using the Kolmogorov-Smirnov nonparametric measure of difference between The labeling we seek is, by definition, the minimum of the distributions of these features. Large disparities encourage inter- this energy. vening boundaries and distinct partition labels. The number of model The partition labels do not classify. Instead they are parameters is minimized by forbidding label configurations that are in- generic and are assigned to blocks of pixels; the size of consistent with prior beliefs, such as those defining very small regions, the blocks (or label resolution) is variable, and depends or redundant or blindly ending boundary placements.
    [Show full text]
  • Revisiting Visual Question Answering Baselines 3
    Revisiting Visual Question Answering Baselines Allan Jabri, Armand Joulin, and Laurens van der Maaten Facebook AI Research {ajabri,ajoulin,lvdmaaten}@fb.com Abstract. Visual question answering (VQA) is an interesting learning setting for evaluating the abilities and shortcomings of current systems for image understanding. Many of the recently proposed VQA systems in- clude attention or memory mechanisms designed to support “reasoning”. For multiple-choice VQA, nearly all of these systems train a multi-class classifier on image and question features to predict an answer. This pa- per questions the value of these common practices and develops a simple alternative model based on binary classification. Instead of treating an- swers as competing choices, our model receives the answer as input and predicts whether or not an image-question-answer triplet is correct. We evaluate our model on the Visual7W Telling and the VQA Real Multiple Choice tasks, and find that even simple versions of our model perform competitively. Our best model achieves state-of-the-art performance on the Visual7W Telling task and compares surprisingly well with the most complex systems proposed for the VQA Real Multiple Choice task. We explore variants of the model and study its transferability between both datasets. We also present an error analysis of our model that suggests a key problem of current VQA systems lies in the lack of visual grounding of concepts that occur in the questions and answers. Overall, our results suggest that the performance of current VQA systems is not significantly arXiv:1606.08390v2 [cs.CV] 22 Nov 2016 better than that of systems designed to exploit dataset biases.
    [Show full text]
  • Some Statistical Models for Prediction Jonathan Auerbach Submitted In
    Some Statistical Models for Prediction Jonathan Auerbach Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy under the Executive Committee of the Graduate School of Arts and Sciences COLUMBIA UNIVERSITY 2020 © 2020 Jonathan Auerbach All Rights Reserved Abstract Some Statistical Models for Prediction Jonathan Auerbach This dissertation examines the use of statistical models for prediction. Examples are drawn from public policy and chosen because they represent pressing problems facing U.S. governments at the local, state, and federal level. The first five chapters provide examples where the perfunctory use of linear models, the prediction tool of choice in government, failed to produce reasonable predictions. Methodological flaws are identified, and more accurate models are proposed that draw on advances in statistics, data science, and machine learning. Chapter 1 examines skyscraper construction, where the normality assumption is violated and extreme value analysis is more appropriate. Chapters 2 and 3 examine presidential approval and voting (a leading measure of civic participation), where the non-collinearity assumption is violated and an index model is more appropriate. Chapter 4 examines changes in temperature sensitivity due to global warming, where the linearity assumption is violated and a first-hitting-time model is more appropriate. Chapter 5 examines the crime rate, where the independence assumption is violated and a block model is more appropriate. The last chapter provides an example where simple linear regression was overlooked as providing a sensible solution. Chapter 6 examines traffic fatalities, where the linear assumption provides a better predictor than the more popular non-linear probability model, logistic regression.
    [Show full text]
  • Visual Question Answering
    Visual Question Answering Sofiya Semenova I. INTRODUCTION VQA dataset [1] and the Visual 7W dataset [20], The problem of Visual Question Answering provide questions in both multiple choice and (VQA) offers a difficult challenge to the fields open-ended formats. Several models, such as [8], of computer vision (CV) and natural language provide several variations of their system to solve processing (NLP) – it is jointly concerned with more than one question/answer type. the typically CV problem of semantic image un- Binary. Given an input image and a text-based derstanding, and the typically NLP problem of query, these VQA systems must return a binary, semantic text understanding. In [29], Geman et “yes/no" response. [29] proposes this kind of VQA al. describe a VQA-based “visual turing test" for problem as a “visual turing test" for the computer computer vision systems, a measure by which vision and AI communities. [5] proposes both a researchers can prove that computer vision systems binary, abstract dataset and system for answering adequately “see", “read", and reason about vision questions about the dataset. and language. Multiple Choice. Given an input image and a text- The task itself is simply put - given an input based query, these VQA systems must choose the image and a text-based query about the image correct answer out of a set of answers. [20] and [8] (“What is next to the apple?" or “Is there a chair are two systems that have the option of answering in the picture"), the system must produce a natural- multiple-choice questions. language answer to the question (“A person" or “Fill in the blank".
    [Show full text]
  • Craftassist: a Framework for Dialogue-Enabled Interactive Agents
    CraftAssist: A Framework for Dialogue-enabled Interactive Agents Jonathan Gray * Kavya Srinet * Yacine Jernite Haonan Yu Zhuoyuan Chen Demi Guo Siddharth Goyal C. Lawrence Zitnick Arthur Szlam Facebook AI Research fjsgray,[email protected] Abstract This paper describes an implementation of a bot assistant in Minecraft, and the tools and platform allowing players to interact with the bot and to record those interactions. The purpose of build- ing such an assistant is to facilitate the study of agents that can complete tasks specified by dia- logue, and eventually, to learn from dialogue in- teractions. 1. Introduction Figure 1. An in-game screenshot of a human player using in-game While machine learning (ML) methods have achieved chat to communicate with the bot. impressive performance on difficult but narrowly-defined tasks (Silver et al., 2016; He et al., 2017; Mahajan et al., 2018; Mnih et al., 2013), building more general systems that perform well at a variety of tasks remains an area of Longer term, we hope to build assistants that interact and active research. Here we are interested in systems that collaborate with humans to actively learn new concepts and are competent in a long-tailed distribution of simpler tasks, skills. However, the bot described here should be taken as specified (perhaps ambiguously) by humans using natural initial point from which we (and others) can iterate. As the language. As described in our position paper (Szlam et al., bots become more capable, we can expand the scenarios 2019), we propose to study such systems through the de- where they can effectively learn.
    [Show full text]
  • Kavita Ramanan; Nicolai Society for Industrial and Applied Meinshausen Mathematics (SIAM) in 2010
    Volume 44 • Issue 4 IMS Bulletin June/July 2015 Donald Geman elected to NAS The USA’s National Academy of Sciences elected 84 new members and 21 foreign CONTENTS associates from 15 countries this year, in recognition of their distinguished and continuing 1 NAS elects Don Geman achievements in original research. Among them was IMS Fellow Donald Geman, who is Professor in the Department of Applied Mathematics and Statistics at Johns Hopkins 2 Members’ News: AMS Fellows, Alison Etheridge, University, Baltimore, MD, and simultaneously a visiting professor at École Normale Emery Brown Supérieure de Cachan. Donald Geman was born in Chicago in 1943. He graduated from the University 3 Carver Award: Patrick Kelly of Illinois at Urbana-Champaign in 1965 with a BA in English Literature and from 4 IMS Travel Awards Northwestern University in 1970 with a PhD in Mathematics. He worked as a 5 Puzzle deadline extended; Professor in the Department of IMU Itô travel awards Mathematics and Statistics at the University of Massachusetts 6 IMS Fellows following graduation, until he 8 Obituary: Evarist Giné joined Johns Hopkins University 9 Recent papers: AIHP in 2001. Donald was elected a Fellow 10 Medallion Lectures: Grégory Miermont; of IMS in 1997, and of the Kavita Ramanan; Nicolai Society for Industrial and Applied Meinshausen Mathematics (SIAM) in 2010. He gave an IMS Medallion Lecture 13 Treasurer’s Report in 2012 at JSM San Diego, on 15 Terence’s Stuff: Ideas “Order Statistics and Gene Regulation.” (See photo.) Donald Geman (right) gave an IMS Medallion Lecture in 2012 16 IMS meetings at JSM in San Diego.
    [Show full text]
  • Shape Quantization and Recognition with Randomized Trees
    Communicated by Shimon Ullman Shape Quantization and Recognition with Randomized Trees Yali Amit Department of Statistics, University of Chicago, Chicago, IL, 60637, U.S.A. Donald Geman Department of Mathematics and Statistics, University of Massachusetts, Amherst, MA 01003, U.S.A. We explore a new approach to shape recognition based on a virtually infi- nite family of binary features (queries) of the image data, designed to ac- commodate prior information about shape invariance and regularity.Each query corresponds to a spatial arrangement of several local topographic codes (or tags), which are in themselves too primitive and common to be informative about shape. All the discriminating power derives from relative angles and distances among the tags. The important attributes of the queries are a natural partial ordering corresponding to increasing structure and complexity; semi-invariance, meaning that most shapes of a given class will answer the same way to two queries that are successive in the ordering; and stability, since the queries are not based on distin- guished points and substructures. No classifier based on the full feature set can be evaluated, and it is impossible to determine a priori which arrangements are informative. Our approach is to select informative features and build tree classifiers at the same time by inductive learning. In effect, each tree provides an approximation to the full posterior where the features chosen depend on the branch that is traversed. Due to the number and nature of the queries, standard decision tree construction based on a fixed-length feature vec- tor is not feasible. Instead we entertain only a small random sample of queries at each node, constrain their complexity to increase with tree depth, and grow multiple trees.
    [Show full text]
  • Visual Coreference Resolution in Visual Dialog Using Neural Module Networks
    Visual Coreference Resolution in Visual Dialog using Neural Module Networks Satwik Kottur1,2⋆, Jos´eM. F. Moura2, Devi Parikh1,3, Dhruv Batra1,3, and Marcus Rohrbach1 1 Facebook AI Research, Menlo Park, USA 2 Carnegie Mellon University, Pittsburgh, USA 3 Georgia Institute of Technology, Atlanta, USA Abstract. Visual dialog entails answering a series of questions grounded in an image, using dialog history as context. In addition to the challenges found in visual question answering (VQA), which can be seen as one- round dialog, visual dialog encompasses several more. We focus on one such problem called visual coreference resolution that involves determin- ing which words, typically noun phrases and pronouns, co-refer to the same entity/object instance in an image. This is crucial, especially for pronouns (e.g., ‘it’), as the dialog agent must first link it to a previous coreference (e.g., ‘boat’), and only then can rely on the visual grounding of the coreference ‘boat’ to reason about the pronoun ‘it’. Prior work (in visual dialog) models visual coreference resolution either (a) implic- itly via a memory network over history, or (b) at a coarse level for the entire question; and not explicitly at a phrase level of granularity. In this work, we propose a neural module network architecture for visual dialog by introducing two novel modules—Refer and Exclude—that per- form explicit, grounded, coreference resolution at a finer word level. We demonstrate the effectiveness of our model on MNIST Dialog, a visually simple yet coreference-wise complex dataset, by achieving near perfect accuracy, and on VisDial, a large and challenging visual dialog dataset on real images, where our model outperforms other approaches, and is more interpretable, grounded, and consistent qualitatively.
    [Show full text]
  • BMC Bioinformatics Biomed Central
    BMC Bioinformatics BioMed Central Research article Open Access The ordering of expression among a few genes can provide simple cancer biomarkers and signal BRCA1 mutations Xue Lin1, Bahman Afsari2, Luigi Marchionni3, Leslie Cope3, Giovanni Parmigiani3,4, Daniel Naiman*1 and Donald Geman1,5 Address: 1Department of Applied Mathematics and Statistics, The Johns Hopkins University, Baltimore, Maryland, USA, 2Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, Maryland, USA, 3Department of Oncology, The Johns Hopkins Kimmel Cancer Center, Baltimore, Maryland, USA, 4Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA and 5Institute for Computational Medicine, The Johns Hopkins University, Baltimore, Maryland, USA Email: Xue Lin - [email protected]; Bahman Afsari - [email protected]; Luigi Marchionni - [email protected]; Leslie Cope - [email protected]; Giovanni Parmigiani - [email protected]; Daniel Naiman* - [email protected]; Donald Geman - [email protected] * Corresponding author Published: 20 August 2009 Received: 12 May 2009 Accepted: 20 August 2009 BMC Bioinformatics 2009, 10:256 doi:10.1186/1471-2105-10-256 This article is available from: http://www.biomedcentral.com/1471-2105/10/256 © 2009 Lin et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract Background: A major challenge in computational biology is to extract knowledge about the genetic nature of disease from high-throughput data. However, an important obstacle to both biological understanding and clinical applications is the "black box" nature of the decision rules provided by most machine learning approaches, which usually involve many genes combined in a highly complex fashion.
    [Show full text]
  • Two-Transcript Gene Expression Classifiers in the Diagnosis And
    BMC Genomics BioMed Central Research article Open Access Two-transcript gene expression classifiers in the diagnosis and prognosis of human diseases LucasBEdelman1,2,6, Giuseppe Toia1,2, Donald Geman4, Wei Zhang5 and Nathan D Price*1,2,3 Address: 1Institute for Genomic Biology, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801, USA, 2Department of Bioengineering, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801, USA, 3Department of Chemical and Biomolecular Engineering, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801, USA, 4Department of Applied Mathematics and Statistics & Institute for Computational Medicine, Johns Hopkins University, Baltimore, Maryland 21218, USA, 5Department of Pathology, University of Texas MD Anderson Cancer Center, Houston, Texas 77030, USA and 6Babraham Institute, Cambridge, CB22 3AT, UK Email: Lucas B Edelman - [email protected]; Giuseppe Toia - [email protected]; Donald Geman - [email protected]; Wei Zhang - [email protected]; Nathan D Price* - [email protected] * Corresponding author Published: 5 December 2009 Received: 13 May 2009 Accepted: 5 December 2009 BMC Genomics 2009, 10:583 doi:10.1186/1471-2164-10-583 This article is available from: http://www.biomedcentral.com/1471-2164/10/583 © 2009 Edelman et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract Background: Identification of molecular classifiers from genome-wide gene expression analysis is an important practice for the investigation of biological systems in the post-genomic era - and one with great potential for near-term clinical impact.
    [Show full text]
  • Resolving Vision and Language Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenes
    Resolving Vision and Language Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenes Gordon Christie1;∗, Ankit Laddha2;∗, Aishwarya Agrawal1, Stanislaw Antol1 Yash Goyal1, Kevin Kochersberger1, Dhruv Batra1 1Virginia Tech 2Carnegie Mellon University [email protected] fgordonac,aish,santol,ygoyal,kbk,[email protected] Abstract We present an approach to simultaneously perform semantic segmentation and prepositional phrase attachment resolution for captioned images. Some ambigu- ities in language cannot be resolved without simultaneously reasoning about an associated image. If we consider the sentence “I shot an elephant in my paja- mas”, looking at language alone (and not using common sense), it is unclear if it is the person or the elephant wearing the pajamas or both. Our approach pro- duces a diverse set of plausible hypotheses for both semantic segmentation and prepositional phrase attachment resolution that are then jointly re-ranked to select the most consistent pair. We show that our semantic segmentation and preposi- tional phrase attachment resolution modules have complementary strengths, and that joint reasoning produces more accurate results than any module operating in isolation. Multiple hypotheses are also shown to be crucial to improved multiple- module reasoning. Our vision and language approach significantly outperforms the Stanford Parser [1] by 17.91% (28.69% relative) and 12.83% (25.28% relative) in two different experiments. We also make small improvements over DeepLab- CRF [2]. Keywords: semantic segmentation; prepositional phrase ambiguity resolution * The first two authors contributed equally Preprint submitted to Computer Vision and Image Understanding September 27, 2017 1. Introduction Perception and intelligence problems are hard. Whether we are interested in understanding an image or a sentence, our algorithms must operate under tremen- dous levels of ambiguity.
    [Show full text]