LREC 2020 Workshop Language Resource and Evaluation Conference 11–16 May 2020 1st International Workshop on Artificial Intelligence for Historical Image Enrichment and Access (AI4HI-2020) PROCEEDINGS Editors: Yalemisew Abgaz, Adapt Centre, Dublin City University Amelie Dorn, Austrian Academy of Sciences, Vienna, Austria Jose Luis Preza Diaz, Austrian Academy of Sciences, Vienna, Austria Gerda Koch, AIT Forschungsgesellschaft mbH, Europeana Local AT Proceedings of the LREC 2020 First International workshop on Artificial Intelligence for Historical Image Enrichment and Access (AI4HI-2019) Edited by: Yalemisew Abgaz, Amelie Dorn, Jose Luis Preza Diaz, and Gerda Koch Acknowledgements: The workshop is supported by the ChIA Project funded by the go!digital programme of the Austrian Academy of Sciences and the ADAPT Centre at Dublin City University. ISBN: 979-10-95546-63-4 EAN: 9791095546634 For more information: European Language Resources Association (ELRA) 9 rue des Cordelières 75013, Paris France http://www.elra.info Email: [email protected] c European Language Resources Association (ELRA) These Workshop Proceedings are licensed under a Creative Commons Attribution-NonCommercial 4.0 International License ii Preface AI4HI-2020 is a major forum that brings together participants from various disciplines to present, discuss, disseminate and share insights on the exploitation of Artificial Intelligence (AI) techniques, semantic web technologies and language resources for the semantic enrichment, search and retrieval of cultural images for the first time. The workshop is co-located with the 12th Edition of Language Resources and Evaluation Conference at the Palais du Pharo, Marseille, France. Semantic web technologies are capable of enriching the data with the required semantics; however, existing ontologies and available models do not fully support the domain-specific requirements of users. The workshop attracts significant attention to make historical images accessible to the general public, as more domain-specific semantics becomes available. The AI4HI-2020 workshop has a general focus on the application of artificial intelligence, semantic web technologies such as ontologies, thesauri and controlled vocabularies, and language resources to enrich and improve access to images related (but not limited) to historical and cultural heritage. This workshop provides the platform to discuss research results including experiments, use cases, experiences, best practices, methods and recommendations for the use of AI and semantic web technologies for historical images. The workshop attracted papers from many stakeholders including AI researchers, NLP experts, digital humanists, linguists, computer scientists and ontology engineers together to present their work and share their experiences. Y. Abgaz, A. Dorn, J. L. Preza Diaz, G. Koch 11-16 May 2020 iii Organising Committee Yalemisew Abgaz, Adapt Centre, Dublin City University • Amelie Dorn, Austrian Academy of Sciences, Vienna, Austria • Jose Luis Preza Diaz, Austrian Academy of Sciences, Vienna, Austria • Gerda Koch, AIT Forschungsgesellschaft mbH, Europeana Local AT • Program Committee Renato Rocha Souza, Austrian Academy of Sciences, Vienna, Austria • Anna Fensel, Semantic Technology Institute (STI), University of Innsbruck, Austria • John Roberto, Adapt Centre, Dublin City University, Dublin, Ireland • Nicole High-Steskal, Department for Image Science, Danube University Krems • Chao-Hong Liu, Dublin City University, Dublin, Ireland • iv Table of Contents Enriching Historic Photography with Structured Data using Image Region Segmentation Taylor Arnold and Lauren Tilton . .1 Interlinking Iconclass Data with Concepts of Art & Architecture Thesaurus AnnaBreit............................................................................. 11 Toward the Automatic Retrieval and Annotation of Outsider Art images: A Preliminary Statement John Roberto, Diego Ortego and Brian Davis . 16 Automatic Matching of Paintings and Descriptions in Art-Historic Archives using Multimodal Analysis Christian Bartz, Nitisha Jain and Ralf Krestel . 23 Towards a Comprehensive Assessment of the Quality and Richness of the Europeana Metadata of food- related Images Yalemisew Abgaz, Amelie Dorn, Jose Luis Preza Diaz and Gerda Koch . 29 v Workshop Program 14:30–16:00 Session 1 14:30–14:40 Welcoming Speech 14:40–15:10 Enriching Historic Photography with Structured Data using Image Region Segmen- tation Taylor Arnold and Lauren Tilton 15:10–15:30 Interlinking Iconclass Data with Concepts of Art & Architecture Thesaurus Anna Breit 14:30–16:00 Session 2 14:30–15:00 Toward the Automatic Retrieval and Annotation of Outsider Art images: A Prelimi- nary Statement John Roberto, Diego Ortego and Brian Davis 15:00–15:15 Automatic Matching of Paintings and Descriptions in Art-Historic Archives using Multimodal Analysis Christian Bartz, Nitisha Jain and Ralf Krestel 15:15–15:30 Towards a Comprehensive Assessment of the Quality and Richness of the Europeana Metadata of food-related Images Yalemisew Abgaz, Amelie Dorn, Jose Luis Preza Diaz and Gerda Koch 15:30–15:50 Concluding Remarks and Next Step vi Proceedings of the 1st International Workshop on Artificial Intelligence for Historical Image Enrichment and Access (AI4HI-2020), pages 1–10 Language Resources and Evaluation Conference (LREC 2020), Marseille, 11–16 May 2020 c European Language Resources Association (ELRA), licensed under CC-BY-NC Enriching Historic Photography with Structured Data using Image Region Segmentation * Taylor Arnold†, Lauren Tilton †Department of Mathematics and Computer Science, University of Richmond *Department of Rhetoric and Communication Studies, University of Richmond 410 Westhampton Way, Richmond, VA, USA 23173 tarnold2, ltilton @richmond.edu { } Abstract Cultural institutions such as galleries, libraries, archives and museums continue to make commitments to large scale digitization of collections. An ongoing challenge is how to increase discovery and access through structured data and the semantic web. In this paper we describe a method for using computer vision algorithms that automatically detect regions of “stuff”—such as the sky, water, and roads—to produce rich and accurate structured data triples for describing the content of historic photography. We apply our method to a collection of 1610 documentary photographs produced in the 1930s and 1940 by the FSA-OWI division of the U.S. federal government. Manual verification of the extracted annotations yields an accuracy rate of 97.5%, compared to 70.7% for relations extracted from object detection and 31.5% for automatically generated captions. Our method also produces a rich set of features, providing more unique labels (1170) than either the captions (1040) or object detection (178) methods. We conclude by describing directions for a linguistically-focused ontology of region categories that can better enrich historical image data. Open source code and the extracted metadata from our corpus are made available as external resources. Keywords: computer vision, image segmentation, cultural heritage, photography, Linked Data, ontology, digital humanities 1. Introduction serves as a roadblock to providing rich links between and across collections, as well as limiting the possibilities for Galleries, libraries, archives, and museums (known as large-scale analysis. While expert and crowd-sourced an- GLAM institutions) and other cultural heritage organiza- notations can fill in some gaps, manual data construction tions have increasingly sought to provide structured meta- requires extensive resources and becomes more difficult as data about historic collections in an effort to increase ac- digitized datasets increase in size (Seitsonen, 2017). cess and discovery. Where records have been digitized and Computer vision techniques provide a direction for the au- rights restrictions allow for it, many of these organizations tomated creation of structured data to enrich collections of have also been able to make the digital records directly ac- historic digital images. Machine learning techniques can cessible through openly available APIs and URIs. Promi- detect features present in images and store these alongside nent examples of these efforts include the Rijksmuseum’s human-generated metadata pertaining to the digital records. RijksData (Dijkshoorn et al., 2018), Europeana’s Search However, the use of automated techniques have their own API, Record API, and SPARQL endpoint (Concordia et al., unique set of challenges. Most computer vision algorithms 2009), and the Linked Data Service provided by the United are built using modern datasets, and may produce annota- States Library of Congress (Zimmer, 2015). The effort to tions that are inaccurate or inappropriate for historic data. make resources available within a cohesive semantic web Incorrectly extracted data records are particularly concern- offers exciting possibilities for research and public access ing when making data available to the public. Even when to cultural collections. Yet, challenges remain for produc- including confidence scores for extracted features, studies ing structured data that facilitates access and exploration of have shown that people have trouble accurately interpreting digital archives. probabilistic data and are overly confident in predictions Many digital collections held by cultural heritage organiza- (Khaw et al., 2019). The challenges of mis-classified data tions consist of still and moving image data. These include are particularly acute when