A Case Study of NLG from Multimedia Data Sources: Generating Architectural Landmark Descriptions
Simon Mille1 , Spyridon Symeonidis2 , Maria Rousi2, Montserrat Marimon Felipe1, Klearchos Stavrothanasopoulos 2, Petros Alvanitopoulos2, Roberto Carlini1, Jens Grivolla1, Georgios Meditskos2, Stefanos Vrochidis2, and Leo Wanner1,3
1Universitat Pompeu Fabra, Barcelona, Spain Corresponding author UPF: [email protected] 2Information Technologies Institute - CERTH, Thessaloniki, Greece Corresponding author ITI-CERTH: [email protected] 3Catalan Institute for Research and Advanced Studies (ICREA)
Abstract et al., 2015). A few others merge multimedia con- In this paper, we present a pipeline system tent in the context of other tasks such as retrieval; that generates architectural landmark descrip- cf. (Clinchant et al., 2011). In most of these works, tions using textual, visual and structured data. the integration is done using similarity measures The pipeline comprises five main components: in a multimedia vector space. To the best of our (i) a textual analysis component, which ex- knowledge, none aims for integration (or fusion) tracts information from Wikipedia pages; (ii) at, and subsequent generation from, the level of on- a visual analysis component, which extracts tological
3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), Dublin, Ireland (Virtual), 18 December 2020, pages 2–14, c 2020 Association for Computational Linguistics Attribution 4.0 International. Petronas Towers location Kuala Lumpur ery new domain, style, language, etc. Machine Petronas Towers restorationDate 2001 Petronas Towers floorArea 395,000 learning-based generators have the best coverage, Petronas Towers floorCount 88 but the relevance and the quality of the produced Petronas Towers elevatorCount 40 texts cannot be ensured. Furthermore, they are fully Petronas Towers buildingType commercial offices Petronas Towers buildingType tourist attraction dependent on the available (still scarce and mostly Petronas Towers highestRegion world monolingual) training data. The development of Petronas Towers highestStart 1998 Petronas Towers highestEnd 2004 grammar-based generators is time-consuming and Petronas Towers architect Cesar´ Pelli they usually have coverage issues. However, they do not require training material, allow for a greater Table 1: A set of
3 6 features per landmark can be obtained through DBpedia (up to 13 for a single landmark). The information that corresponds to one feature can be encoded by a variety of property names (up to 10). For instance, the cost of a building can be expressed by dbo:cost, dbp:cost, or dbp:constructionCost.3 The 39 features and the corresponding 98 proper- ties are listed in Table7, Appendix A.2. Figure 1: System architecture. For the retrieval of the corresponding DBpedia triples, we developed a component that applies tains, e.g., the date of creation, the location, the SPARQL queries to the DBpedia SPARQL end- 4 architecture style or the popularity. point. In some rare cases, the queries returned an To create the sufficiently diverse datasets, we error message at the time they were performed; in first manually compiled a list of 160 landmarks such a case, the property cannot be accessed and that vary in terms of the aforementioned character- the information is not retrieved. istics. In the next step, we retrieved the available multimedia content on these landmarks: images 4.2 Visual content acquisition (from Flickr), textual descriptions (from Wikipedia) The performed visual analysis for the purpose of and ontological properties (from DBpedia)1 and visual content acquisition is twofold. First, an ob- selected then a subset of 120 landmarks that had ject detection module classifies indoor and outdoor either rich image or DBpedia contents, or both. We scenes and detects landmark (in this case, building) used 101 landmarks to develop and optimise our elements, and objects. Second, an architectural framework, whereas 19 randomly selected land- style classification module assigns the related ar- marks were left aside for the evaluation stage. The chitectural style label to each outdoor scene of the full list of landmarks is available in Appendix A.1. selected dataset. Both classification modules are based on state-of-the-art deep learning techniques. 4 Multimedia content acquisition 4.2.1 Visual scene classification and labeling In this section, we describe the content that we ex- tracted from the three sources (DBpedia, images For visual scene classification, we draw upon the and texts) used for generating the landmark descrip- 145 relevant indoor and outdoor scene classes from tions, and how we extracted it. the Places dataset (Zhou et al., 2018), which con- tains 1,803,460 images, annotated with a total of 4.1 DBpedia triple retrieval 365 classes. The classifier is a VGG16 deep neu- DBpedia contains a lot of information that is po- ral network (Simonyan and Zisserman, 2014), pre- tentially relevant to the description of architectural trained on the Places dataset for the first 14 layers landmarks. We analysed manually the DBpedia (for all of its 365 classes) and fine-tuned on a subset entries of the 101 landmarks in the development of 145 selected classes for the last two layers. set in order to see which properties are related to For labeling the landmark elements and objects the landmark and its architectural features.2 We in the classified scenes, we use a Deeplab model, identified 39 features of interest, most of which are pre-trained on the PASCAL VOC (Everingham consistently found across the landmarks in the list, et al., 2009) dataset, where further training was among them, e.g., features related to the type of applied using a combination of building fac¸ade the landmark, its style, who built it, the dates of its segmentation datasets (Mapillary Vistas (Neuhold construction, renovation, or extension, its location, et al., 2017), CMP (Tylecek and Sara´ , 2013), ECP,5 its construction materials, its cost, its number of LabelMeFacade (Frohlich et al., 2010), eTRIMS floors, elevators, or towers, etc. On average, about 3There are two main types of properties in DBpedia: 1While the resources on Wikipedia and DBpedia are free “clean” properties that stem from the DBpedia ontology (dbo: for use, we had to pay special attention to image collection prefix), and properties automatically extracted from raw from Flickr to ensure that we gather media whose license Wikipedia infoboxes (dbp: prefix). permits their reuse for our purposes. 4http://dbpedia.org/sparql 2See for illustration the Petronas Towers page: http: 5http://vision.mas.ecp.fr/Personnel/ //dbpedia.org/page/Petronas_Towers. teboul/data.php
4 (Korcˇ and Forstner¨ , 2009)).6 This not only resulted ones used for text generation as such. in a computationally efficient implementation for A list of the supported Architectural styles the detection of architectural landmark-related arte- Art Deco Art Nouveau Baroque Bauhaus facts, but also increased the classification accuracy Biedermeier Corinthian Order Deconstructuvism Doric Order of the model. In order to further improve object Early Roman Gothic Hellinistic Ionic Order Modernist Neoclassical Postmodernism Renaisance detection, we added a third module based on the Rococo Romanesque Mask RCNN model. We initialized this module using pre-trained weights on COCO dataset (Lin Table 2: Architectural Styles et al., 2014) and then performed fine-tuning on a customized set created by merging LVIS (Gupta For training of the model for landmark style and et al., 2019) and ADEK20K (Zhou et al., 2016) feature identification, images from Flickr, Euro- datasets and removing all classes irrelevant to the peana and Wiki were collected. Annotators with scope of our task. Details about the training set- architectural expertise annotated the collected data. tings are provided in Appendix A.3. A set of 11,368 newly annotated images was used for training purposes, while a total of 1,276 newly annotated images were used for testing. VGG16 and ResNet50 models were enhanced with 3 layers (one GlobalAveragePooling2D and two Dense lay- ers) and initialised with the pre-trained ImageNet weights. For better training, K-Fold Validation and Stratified Shuffle Split were applied.
4.3 Textual content acquisition Figure 2: (L)Visual of the detection module’s results. The algorithm detects the building along with the sur- In addition to using visual features, we enriched roundings and generates the corresponding predicted the DBpedia information by entity-relation-entity tags. (R) Architecture style recognition, the model pre- triples extracted from the unstructured part of the dicted label : Romanesque , True label : Romanesque. Wikipedia articles. This is done using a pipeline that comprises concept detection, entity linking and WSD to identify the proper entities (linking to DB- 4.2.2 Landmark style identification pedia URIs); then, PoS tagging, dependency pars- In the context of this task, the visual analysis com- ing and coreference resolution to generate surface- ponent aims to assign to a landmark one of the 18 syntactic structures and to link mentions of entities architectural styles listed in Table2 and to iden- in the different parts of text; and, finally, semantic tify the following seven additional types of features parsing to generate deep-syntactic structures from that are later on used for text generation: (i) con- which we extract the triples. struction type (e.g., ‘amphitheater’, ‘castle’, ‘ho- In what follows, we outline the types of infor- tel’); (ii) similarities with other construction types mation we aim to extract from the textual data and (same list as (i)); (iii) similarity of a part of the con- how we do it. struction with another construction (e.g., ‘bridge’, ‘arch’); (iv) facade elements (e.g., ‘balcony’, ‘fire 4.3.1 Targeted information in textual data escape’); (v) interior components and objects (e.g., Unlike the content extracted from visuals, which ‘fireplace’, ‘elevator shaft’, etc.); (vi) environment is not expected to be found in DBpedia since it is (e.g., ‘downtown’, ‘village’, ‘park’); (vii) proxim- related to specific images and some “subjective” ity to a natural landmark (e.g., ‘river’, ‘park’). The features, the information extracted from Wikipedia full list of detected features is provided in Table8, is supposed to be already captured in DBpedia. Appendix A.3. In total, eight different properties However, we observe that some of the relevant are extracted for generation, two of which (style, properties are often missing. The goal of the textual construction type) are fused with the properties ob- analysis component is thus to recover these missing tained through the other modalities, and the other properties, which concern, in particular, the type of the landmark, its date(s) of construction and 6These datasets contain up to 25,000 high-resolution im- ages annotated with a variety of semantic classes and possibly renovation, its location, its architectural style and instance-specific labels. its architect, designer or creator. In order to reduce
5 the load on textual analysis, we analyse only the extraction for the targeted triples (see Section 4.3). first paragraphs of the scraped Wikipedia articles. In other words, the coverage of the module needs As an example, consider the text “Rouen Cathe- to be extended to get more information, and some dral is a Roman Catholic church in Rouen, Nor- incorrect values would need to be filtered. The mandy, France. It is the seat of the Archbishop of main limitations here are the difficulty in covering Rouen, Primate of Normandy. The cathedral is in the wide variety of surface syntactic structures, and the Gothic architectural tradition.”, for which the the quality of syntactic parses. following triples are extracted: For the evaluation of the architectural style clas- sification model a set of building images were se- Rouen Cathedral Localisation Roman Catholic church Rouen Cathedral Location Rouen, Normandy, France lected. The dataset for testing comprises 1,276 Rouen Cathedral Style Gothic architectural tradition images and includes all the 18 architectural styles. The F1 score was taken into consideration and a significant 46.16% of correct classification was per- 4.3.2 Triple extraction from texts formed, similar to the SoA results (Z. et al., 2014); In order to extract the targeted triples, we apply a the confusion matrix for the architecture style clas- sequence of rule-based graph transducers on the sification is shown in Figure7 in Appendix A.3. output of an off-the-shelf syntactic parser. More Even though the classification is state-of-the-art, in specifically, we run the pipeline used for creating more than 50% of the cases the architectural style the deep input representations of the Surface Re- is wrong, which is one of the main comments from alisation shared tasks 2018 and 2019 (Mille et al., the evaluators in terms of incorrectness of contents 2018), with one additional component responsi- (see Section 7.4). ble of identifying the configurations that corre- spond to the targeted information. Consider, for 5 Multimedia information fusion illustration, a sample rule in Figure3, which ex- tracts the ‘Rouen Cathedral – Localisation – Ro- The results of the visual and textual analyses and man Catholic church’ triple from the predicate- the retrieved DBpedia properties are mapped us- argument (PredArg) structure as encoded by light ing the Web Annotation Data Model7. The model verb be-constructions, where the first argument of creates a body and a target for each annotation. the light verb be becomes the first argument of the As main interconnection point between the con- predicate in the PredArg representation. tent from different media, we use the name of the corresponding entity, which is mapped as the tar- get of the annotation. The body of the annotation contains all the other information, which varies ac- cording to the nature of each input module. More specifically: (i) for the visual analysis content, the body contains information pertinent to scene, ob- jects, fac¸ade, structure elements and architectural styles. A mapping example is shown in Figure4; (ii) for the Wikipedia analysis outcome, the body Figure 3: A sample rule to extract triples. The Left- contains creator and localisation information re- Side matches a part of the input tree, the RightSide lated to the entity; (iii) for the retrieved DBpedia builds part of the output. Three types of objects are triples, the information in the body is pertinent to used: nodes (?N ), relations between nodes (?r ) {} → the landmark, the architecture, the location and and attribute-value pairs associated to a node (?a = more general information about the landmark. b), where question marks indicate variables and text in A reasoning mechanism applies the following black indicates literal strings. property-based semantic rules at the time of the retrieval of DBpedia triples: (i) Extraction of class 4.4 Quantitative analysis of the information information about creators and locations: the mech- acquisition modules. anism detects whether a creator is a person, an or- We evaluated the text analysis component with a ganisation or a company, and whether a location is set of 16 texts and measured average values of 7https://www.w3.org/TR/ 83% for precision and 40% for recall on the triple annotation-model/
6 @prefix examples:
7 of the specificities of our pipeline is that two types not only larger due to the input size, but also more of aggregation take place during generation, one complex due to the more complex syntactic con- at the predicate-argument level (in a NaturalOWL structions (e.g., non-projective trees, as in highest fashion), and one at the syntactic level (see below). building in the world). Finally, we crafted a new syntactic aggregation module, which aggregates 6.2 Extension of an existing generator coordinated and relative clauses based on identity The base generator covers about 400 DBpedia prop- of syntactic subjects/objects, locations and verbs.9 erties, but only a few had to do with architectural landmarks, and generation of up to only 10 triples 7 Evaluation had been tested. In this work, the inputs can contain We evaluate the quality of the generated descrip- up to 19 triples, and most of the properties are new. tions from fused representations, first of all, against We thus extended the coverage of the generator monomodal descriptions generated solely from DB- according to two main aspects: (i) addition of 38 pedia triples. The goal is to assess to what extent manually crafted PredArg templates, (ii) addition architectural landmark descriptions benefit from ad- of domain-specific “semantic” aggregation rules. ditional content from other media. A comparison For (i), the 38 new properties8 were each associ- with Wikipedia texts is also carried out. ated with a new PredArg template; see, for instance, the templates corresponding to the ‘highestEnd’ 7.1 Evaluation method and ‘interiorComponent’ properties in Figure5. In- stantiating the template5 (a) with the values of Six journalists, architects and architectural land- Table1 ([name] = Petronas Towers, [highestEnd] = mark content providers were recruited for the eval- 10 2004) and generating it would result in the sentence uation. They were asked to evaluate descriptions The Petronas Towers were the highest building in with respect to their correctness of form and content the world until 2004. Template5 (b) would be and their level of interest by rating the following 11 realised as There is an elevator shaft in P. Towers. statements on a 6-value Likert scale: Correctness of Form: Independently of Correct- ness of content and Interestingness, (i) the surface form of the text is free of grammatical and spelling errors, (ii) the text is easy to read and understand, and (iii) it flows well. (a) Correctness of Content: Independently of Cor- (b) rectness of form and Interestingness, and using only my current knowledge on the topic, I do not Figure 5: Sample predicate-argument templates identify information that looks obviously incorrect. (a) = DBpedia, (b) = visuals Interestingness: Independently Correctness of For (ii), we designed new aggregation rules to form and content, I find the information provided complement the generic rules already in place, in the text interesting, relevant and inspirational. which are based on the identity between predicates The evaluation test set consisted of the descrip- and/or entities only. In particular, properties that tions of 19 landmarks: 19 descriptions generated involve dates need to be aggregated in a specific from fused multimedia representations, 19 gener- way when there are both a start and an end date. ated from DBpedia triples and the first or second For instance, highestStart and highestEnd as seen paragraph (whichever was the most informative in Table1 trigger a rule that introduces a between: in terms of architectural descriptions) from the the highest building in the world between 1998 and Wikipedia articles on the 19 landmarks in question. 2004. In parallel, some other rules aggregate some For each building, all evaluators were presented properties in priority if found in the input: [style + with the three descriptions, in a random order, and date] [creator + style] [generic rules]. 9 In the case of the ‘interiorComponent’ property, as seen Specific rules (e.g., for linearisation) were also in Figure5, there is no verb at the semantic level; it is only in- improved to cover the generation of texts that are troduced in the syntactic structure. The syntactic aggregation module covers such cases. 832 from DBpedia (7 out of the 39 were already covered; 10All evaluators are fluent in English and familiar with the see Table7, Appendix A.2) , and 6 coming from the visual described landmarks. content analysis (see Section 4.2) 11Answers from 1: strongly disagree to 6: strongly agree.
8 scored each description. That is, each system re- Wikipedia (human) The Sydney Opera House is a multi-venue performing arts ceived 114 ratings for each of the 3 criteria, which centre at Sydney Harbour in Sydney, New South Wales, Aus- we believe makes the evaluation trustworthy. tralia. It is one of the 20th century’s most famous and distinc- tive buildings. DBpedia 7.2 Evaluation against DBpedia-based Sydney Opera House, which Jørn Utzon designed, is a Per- descriptions forming arts center in Sydney. Sydney Opera House, the architectural style of which is Expressionist architecture, was The comparison between the descriptions gener- built between 1 March 1959 and 1973. Its structure is made ated from fused multimedia representations and the of Concrete frame & precast concrete ribbed roof. descriptions generated from DBpedia triples (see Fused Sydney Opera House, which Jørn Utzon designed, is a centre Table4) is shown in Figure6. A 2-tailed Mann- in Sydney. Its structure is made of Concrete frame & pre- Whitney U test indicates that only for Form the cast concrete ribbed roof. An element of the structure is like a bridge. Sydney Opera House has similarities with a difference is not statistically significant at p <0.05. beach house, an amusement park and a museum. Sydney Opera House, the architectural style of which is Deconstruc- 6 tuvism, was built between 1 March 1959 and 1973. DBpedia 4.9 Table 4: Sydney Opera house descriptions (eval set) 4.6 Fusion
4 3.8 two criteria are low, in particular for interestingness 3.4 3.1 (2.7 and 3.1). However, adding information from 2.7 different sources does increase slightly the interest- ingness of the texts, and even their correctness in 2 terms of form, but at the expense of the correctness of the contents. The lower scores obtained for interestingness (when compared to the other two criteria) even 0 for the human-written Wikipedia texts highlight Form Content Interest. the difficulty for short texts to be considered in- Figure 6: Results of the human evaluation teresting and inspirational. But the fact that the automatically generated texts score more than one point lower than Wikipedia shows that the content 7.3 Evaluation against Wikipedia representation of the 44 features is still not suffi- The 6 evaluators were also asked to rate the cient; more content needs to be provided. Other Wikipedia articles. In most cases, one single DBpedia properties such as ‘owner’, ‘tenant’, or Wikipedia paragraph was longer and richer than information related to other landmarks or persons either of our generated descriptions, so the texts such as ‘architecturalStyle (of)’, ‘birthPlace (of)’, are not fully comparable, but our objective has ‘influencedBy’, ‘location (of)’, etc. could augment been to define some upper bound scores for a short the interestingness of the descriptions. text. Wikipedia paragraphs scored 5.5, 5.3 and The low scores of our generator for the correct- 4.4 for the correctness of form, of contents and ness of the contents, in particular for the fused interestingness respectively, that is, 0.6, 1.9 and descriptions (3.4) are due to several causes. First 1.3 points higher than our fused descriptions. We of all, as shown by the scores of the DBpedia-only also asked the evaluators to pick which text they descriptions (3.8), not all the information in DBpe- preferred among the 3 candidates, and interestingly, dia is factually correct, in particular the informa- Wikipedia articles were not always chosen: in 15 tion extracted automatically from Infoboxes: build- cases out of 114, an automatically generated text ings can be assigned types such as “Series”, “Nick- was picked (DBpedia: 4, Fusion: 11). name”or “Mixed-use” (see Tables9 and 11, A.4); construction dates can be irrelevant (“between 2015 7.4 Discussion and 532”); locations sometimes refer to a relative Table4 shows texts from the different sources. Fig- location (e.g., “right”), etc. Second, the informa- ure6 shows that while the scores for the correctness tion extracted from texts and visuals, tasks which of form are rather high for both the fused and DBpe- are traditionally difficult to solve, is also not per- dia descriptions (close to 5), the scores for the other fect (a detailed error analysis is provided in Section
9 4.4); incorrect architectural styles (e.g., Tables 10 Ion Androutsopoulos, Gerasimos Lampouras, and Dim- and 12, A.4) and comparisons between supposedly itrios Galanis. 2013. Generating natural language similar buildings (see Table 10, A.4) were found descriptions from owl ontologies: the naturalowl system. Journal of Artificial Intelligence Research, particularly disconcerting by the evaluators. Fi- 48:671–715. nally, the performance of the fusion component is currently heavily dependent on the cases seen in Anja Belz, Mike White, Dominic Espinosa, Eric Kow, the development dataset. In the development set, in Deirdre Hogan, and Amanda Stent. 2011. The first Surface Realisation Shared Task: Overview and most cases, the selected entities were valuable and evaluation results. In Proceedings of the Generation supported the Text Generation as expected, but in Challenges Session at the 13th European Workshop the evaluation set, many cases had not been seen, on Natural Language Generation (ENLG), pages such that ill-informed decisions were taken, some- 217–226, Nancy, France. times triggering the replacement of a correct value Nadjet Bouayad-Agha, Gerard Casamayor, Simon from DBpedia by an incorrect value from visual or Mille, Marco Rospocher, Horacio Saggion, Luciano textual analysis (see Table4 and Tables9 and 12, Serafini, and Leo Wanner. 2012. From ontology to A.4). A larger development set would be needed nl: Generation of multilingual user-oriented environ- in order to identify more erroneous configurations. mental reports. In International Conference on Ap- plication of Natural Language to Information Sys- Another solution may be more generic strategies to tems, pages 216–221. Springer. foresee the possible mistakes in the inputs. Nadjet Bouayad-Agha, Gerard Casamayor, and Leo 8 Conclusions Wanner. 2014. Natural language generation in the context of the semantic web. Semantic Web, We presented the case of the generation of architec- 5(6):493–513. tural landmark descriptions from ontological struc- tures that contain fused content from visual, textual Stephane´ Clinchant, Julien Ah-Pine, and Gabriela and ontological sources. The evaluation showed Csurka. 2011. Semantic combination of textual and visual information in multimedia retrieval. In Pro- that when compared to descriptions generated from ceedings of the 1st ACM international conference on the DBpedia RDF-triples obtained from textual multimedia retrieval, pages 1–8. material only (i.e., Wikipedia), descriptions that communicate fused content are considered more Pradipto Das, Rohini Kesavan Srihari, , and Jason J. interesting and better in terms of textual quality. Corso. 2013. Translating related words to videos and back through latent topics. In Proc. of WSDM. However, also due to the limited content features that were considered in the experiments, these de- Mark Everingham, Luc Van Gool, Christopher K. I. scriptions cannot compete, in general, with more Williams, John Winn, and Andrew Zisserman. 2009. comprehensive well-written descriptions as encoun- The pascal visual object classes (voc) challenge. In- ternational Journal of Computer Vision tered in Wikipedia. Still, it needs to be taken ac- , 88:303– 308. Printed version publication date: June 2010. count that by far not all architectural landmarks that are of interest from the professional or cultural Bjorn Frohlich, Erik Rodner, and Joachim Denzler. viewpoint are covered by Wikipedia. Fused content 2010. A fast approach for pixelwise labeling of fa- descriptions are then a welcomed solution. cade images. In Proceedings of the 2010 20th Inter- national Conference on Pattern Recognition, ICPR ’10, page 3029–3032, USA. IEEE Computer Soci- Acknowledgements ety. This work was supported by the European Commis- sion in the context of its H2020 Program under the Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017a. Creating train- grant numbers 870930-RIA, 779962-RIA, 825079- ing corpora for micro-planners. In Proceedings RIA, 786731-RIA at Universitat Pompeu Fabra and of the 55th Annual Meeting of the Association for Information Technologies Institute - CERTH. Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada. Association for Computational Linguistics. References Claire Gardent, Anastasia Shimorina, Shashi Narayan, Nayyer Aafag, Ajmal Mian, Wei Liu, Syed Zulqarnain and Laura Perez-Beltrachini. 2017b. The WebNLG Gilani, and Mubarak Shah. 2019. Video description: challenge: Generating text from RDF data. In Pro- A survey of methods, datasets and evaluation met- ceedings of the 10th International Conference on rics. ACM Computing Surveys, 52(6). Natural Language Generation, pages 124–133.
10 Albert Gatt and Emiel Krahmer. 2018. Survey of the In Proceedings of the IEEE International Confer- state of the art in natural language generation: Core ence on Computer Vision (ICCV). tasks, applications and evaluation. Journal of Artifi- cial Intelligence Research, 61:65–170. Mick O’Donnell, Chris Mellish, Jon Oberlander, and Alistair Knott. 2001. Ilex: an architecture for a dy- Agrim Gupta, Piotr Dollar,´ and Ross Girshick. 2019. namic hypertext generation system. Natural Lan- Lvis: A dataset for large vocabulary instance seg- guage Engineering, 7(3):225. mentation. Karen Simonyan and Andrew Zisserman. 2014. Very Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratud- deep convolutional networks for large-scale image din, and Hamid Laga. 2019. A comprehensive sur- recognition. vey of deep learning for image captioning. ACM Computing Surveys, 51(6). Radim Tylecek and Radim Sara.´ 2013. Spatial pat- tern templates for recognition of objects with regular Qian Huang, Zhu Liu, Aaron Rosenberg, David Gib- structure. In Pattern Recognition, Lecture Notes in bon, and Behzad Shahraray. 1999. Automated gen- Computer Science, pages 364–374. Springer Berlin eration of news content hierarchy by integrating au- Heidelberg. dio, video, and text information. In 1999 IEEE In- ternational Conference on Acoustics, Speech, and Sebastian Varges and Chris Mellish. 2001. Instance- Signal Processing. Proceedings. ICASSP99 (Cat. No. based natural language generation. In Second Meet- 99CH36258), volume 6, pages 3025–3028. IEEE. ing of the North American Chapter of the Associa- tion for Computational Linguistics. Yanti Idaya Aspura and Shahrul Azman. 2017. Seman- tic text-based image retrieval with multi-modality Leo Wanner, Bernd Bohnet, Nadjet Bouayad-Agha, ontology and dbpedia. The Electronic Library. Francois Lareau, and Daniel Nicklaß. 2010. MAR- QUIS: Generation of user-tailored multilingual air Filip Korcˇ and Wolfgang Forstner.¨ 2009. eTRIMS Im- quality bulletins. Applied Artificial Intelligence, age Database for interpreting images of man-made 24(10):914–952. scenes. Technical Report TR-IGG-P-2009-01. Ran Xu, Caiming Xiong, Wei Chen, and Jason J Corso. Tsung-Yi Lin, Michael Maire, Serge Belongie, James 2015. Jointly modeling deep video and composi- Hays, Pietro Perona, Deva Ramanan, Piotr Dollar,´ tional text to bridge vision and language in a unified and C. Lawrence Zitnick. 2014. Microsoft coco: framework. In Twenty-Ninth AAAI Conference on Common objects in context. In Computer Vision – Artificial Intelligence. ECCV 2014, pages 740–755, Cham. Springer Inter- national Publishing. Xu Z., Tao D., Zhang Y., Wu J., and Tsoi A.C. 2014. Architectural style classification using multinomial Susan W McRoy, Songsak Channarukul, and Syed S latent logistic regression. In Computer Vision – Ali. 2003. An augmented template-based approach ECCV 2014. Lecture Notes in Computer Science, vol to text realization. Natural Language Engineering, 8689, Cham. Springer International Publishing. 9(4):381. Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Simon Mille, Anja Belz, Bernd Bohnet, and Leo Wan- Oliva, and Antonio Torralba. 2018. Places: A 10 ner. 2018. Underspecified Universal Dependency million image database for scene recognition. IEEE Structures as Inputs for Multilingual Surface Real- Transactions on Pattern Analysis and Machine Intel- isation. In Proceedings of the 11th International ligence, 40(6):1452–1464. Conference on Natural Language Generation, pages 199–209, Tilburg, The Netherlands. Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Simon Mille, Stamatia Dasiopoulou, Beatriz Fisas, and 2016. Semantic understanding of scenes through the Leo Wanner. 2019. Teaching FORGe to verbalize ade20k dataset. DBpedia properties in Spanish. In Proceedings of the 12th International Conference on Natural Lan- guage Generation, pages 473–483, Tokyo, Japan. Association for Computational Linguistics. Neha Nayak, Dilek Hakkani-Tur,¨ Marilyn A Walker, and Larry P Heck. 2017. To plan or not to plan? discourse planning in slot-value informed sequence to sequence models for language generation. In Proceedings of INTERSPEECH, pages 3339–3343, Stockholm, Sweden. Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. 2017. The mapillary vistas dataset for semantic understanding of street scenes.
11 A Appendices A.2 List of retrieved DBpedia properties
A.1 List of buildings in the datasets Table7 lists the 39 features used for generation, Tables5 and6 show the buildings used for devel- roughly grouped by topic, and their correspondence opment and evaluation respectively.
Alhambra Arc de Triomphe Features (count) Properties Belem´ Tower Blue Church Bratislava dbo:buildingType, dbo:type, dbp:buildingType, Borobudur Temple Bran Castle building type (54) dbp:type, dbp:architecturalType, Brandenburg Gate Bratislava Castle dbp:architectureType, dbp:category, Buckingham Palace Buda Castle hypernym (89) http://purl.org/linguistics/gold/hypernym Burj Al Arab CN Tower dbo:architecturalStyle, Canton Tower Casa Batllo´ architectural style dbp:architecturalStyle, Casa Mila` Castel Sant’Angelo (49) dbp:architectureStyle, Catherine Palace Chichen Itza dbp:style, dbp:architecture Chrysler Building Chateauˆ Frontenac dbo:architect, dbo:builder, dbp:architect, dbp:author, dbp:builder, dbp:engineer, Chateauˆ de Chenonceau Cologne Cathedral architect (61) Colosseum Dome of the Rock dbp:foundedBy, dbp:renArchitect, Dresden Frauenkirche Edinburgh Castle dbp:renOthDesigners Eiffel Tower Elbphilharmonie architecture firm (2) dbp:architectureFirm Empire State Building Faisal Mosque sculptor (1) dbo:sculptor, dbp:sculptor Fallingwater Fisherman’s Bastion dbo:synonym, dbp:alternateName, dbp:alternateNames, dbp:designation1Offname, Florence Cathedral Forbidden City other name (17) Gatchina Palace Giza Pyramids dbp:designation2Offname, Grand Place Brussels Harpa Concert Hall dbp:nativeName, dbp:otherName Helsinki Cathedral Heydar Aliyev Center former name (2) dbo:formerName Himeji Castle Jin Mao Tower dbo:buildingEndDate, dbp:built, Kiev Pechersk Lavra Knossos Palace dbp:completionDate, dbp:completedDate, Konark Sun Temple Kronborg Castle completion date (45) dbp:dateComplete, dbp:dateConstructionEnds, Lincoln Center Lincoln Memorial dbp:established, dbp:founded, Lloyd’s Building London Eye dbp:used, dbp:yearCompleted Madrid Palace Marina Bay Sands dbo:buildingStartDate, dbo:yearOfConstruction, Metropolitan Cathedral of Bras´ılia Milan Cathedral construction start date dbp:beginningDate, dbp:brokeGround, dbp:date, Mosque of Cordoba´ Musee´ d’Orsay (37) dbp:dateConstructionBegins, dbp:groundbreaking, Niteroi´ Contemporary Art Museum Notre Dame dbp:startDate, dbp:yearsBuilt Odeon of Herodes Atticus One World Trade Center demolition date (1) dbp:demolished Oriental Pearl Tower Palace of Versailles extension date (1) dbp:extension dbp:restored, dbp:dateRenovated, Peles Castle Pena Palace restoration date (4) Petra Jordan Porta Nigra dbp:renovationDate Potala Palace Prague Castle UNESCO designation dbp:year, dbp:whsYear, Rouen Cathedral Royal Liver Building Liverpool date (12) dbp:designation1Date Royal Observatory (Greenwich) Sacre-Coeur´ location (86) dbo:location, dbp:location dbo:country, dbp:country, dbp:locationCountry, Sagrada Familia Cathedral Space Needle country (27) St. Basil’s Cathedral Statue of Liberty dbp:state, dbp:stateParty Stonehenge Sultan Ahmed Mosque culture (2) dbp:cultures Taipei 101 Taj Mahal bell count (4) dbp:bells Tech Tower The Atomium dome count (2) dbp:domeQuantity The Cristo Rei The Flatiron Building elevator count (16) dbp:elevatorCount The Gherkin The Guggenheim New York floor count (18) dbo:floorCount The Lotus Temple The Pantheon Rome minaret count (3) dbp:minaretQuantity The Shard The Sistine Chapel room count (2) dbp:roomCount, dbp:rooms The Temple of Olympian Zeus (Athens) The White House spire count (6) dbp:spireQuantity Tokyo Skytree Tokyo Tower step count (1) dbp:stepCount Tower of Pisa Villa Savoye suite count (1) dbp:suites Wembley Stadium Westminster Abbey tower count (3) dbp:towerQuantity Wilanow´ Palace Windsor Castle cost (17) dbo:cost, dbp:cost, dbp:constructionCost Wuppertal Schwebebahn elevation (1) dbo:elevation floor area (11) dbo:floorArea height (9) dbo:height, dbp:height Table 5: The 101 buildings used for development seating capacity (10) dbp:capacity, dbp:seatingCapacity, dbp:garrison building confused owl:differentFrom with (2) facade direction (3) dbp:facadeDirection Angkor Wat Big Ben highest building dbp:highestStart Burj Khalifa Camp Nou start date (7) Christ the Redeemer Dancing House Prague highest building dbp:highestEnd Guggenheim Museum Bilbao Hagia Sophia end date (7) Hungarian Parliament Building Kremlin highest building dbp:highestRegion Louvre Machu Picchu region (1) Neuschwanstein Castle Parthenon construction dbo:material, dbp:material, Peterhof Palace Petronas Towers material (8) dbp:materials Sydney Opera House Walt Disney Concert Hall structural system (2) dbp:structuralSystem White Tower Thessaloniki Table 7: List of retrieved features from DBpedia, and Table 6: The 19 buildings used for the evaluation number of occurrences in the development set (in grey, properties already covered by the base generator)
12 with the 98 properties from DBpedia. In parenthe- isation as described on the above section. For the ses, the number of times each property had one architectural style recognition task (see the confu- or more value(s) for a building. There can be two sion matrix in Table7), the experiments involved reasons why there are more than one value for a fea- Stochastic Gradient Descent and Adam as opti- ture: (i) one property is given more than one value, misers. Different epochs, batch size and learning or/and (ii) multiple properties have one value. rates were tested. Finally a VGG19 model was Alley alcove amphitheater trained for 130 epochs. The training includes 3-fold amusement park apartment building outdoor aqueduct Arch archaeological excavation atrium public cross validation, and SGD optimiser of learning Attic auditorium balcony exterior balcony interior ball Bar rate equal to 0.001. All trainings and evaluations barn barrier–curb barrier–fence were conducted on a 1080Ti GPU. barrier–guard-rail barrier–wall bathroom bazaar indoor bazaar outdoor beach house bedroom berth bow window indoor A.4 Sample output texts box building facade cafeteria Tables9, 10, 11 and 12 show sample texts for a few campus car case castle catacomb cemetery buildings; the parts of the text that come from the chalet chest of drawers children room church indoor church outdoor classroom textual and visual analysis are shown in bold, and cloister computerroom concert hall incorrect content is shown in red. cornice corridor cottage courthouse courtyard crosswalk Wikipedia (human) deco department store dining hall The Burj Khalifa, known as the Burj Dubai prior to its in- dining room discotheque doorway outdoor downtown driveway eiffel-tower auguration in 2010, is a skyscraper in Dubai, United Arab elevator embassy engine room Emirates. With a total height of 829.8 m (2,722 ft, just over entrance hall escalator indoor excavation half a mile) and a roof height (excluding antenna, but includ- fabric store fac¸ade/wall farm ing a 244 m spire) of 828 m (2,717 ft), the Burj Khalifa has fire escape fire station fireplace been the tallest structure and building in the world since its flat–bike-lane flat–crosswalk-plain flat–curb-cut topping out in 2009 (preceded by Taipei 101). flat–parking flat–pedestrian-area flat–rail-track flat–road flat–sidewalk food court DBpedia formal garden gameroom garage indoor Burj Khalifa, which Adrian Smith (architect) designed, is a garage outdoor gazebo exterior general store indoor Mixed-use in Dubai. It costed 1,500,000,000$. It has 2 floors general store outdoor golden-gate-bridge greenhouse indoor and 57 elevators and a floor area of 309,473m2. It was the greenhouse outdoor gymnasium indoor home office highest building in the world. Burj Khalifa, the architectural home theater hospital hotel outdoor style of which is Neo-futurism, was built between 6 January hotel room house hunting lodge outdoor igloo indoor industrial area 2004 and 31 December 2009. It was built of glass, steel, inn outdoor kasbah kindergarden classroom aluminium, reinforced concrete. It was formerly called Burj kitchen library indoor library outdoor Dubai. living room lighthouse mansion Fused lobby mausoleum manufactured home Burj Khalifa, which Adrian Smith (architect) designed, is a market indoor market outdoor meeting room in Dubai. It costed mirror mosque outdoor motel skyscraper in a downtown environment movie theater indoor oast house museum indoor 1,500,000,000$. It has 2 floors and 57 elevators and a floor museum outdoor nursery office area of 309,473m2. It has similarities with a tower and a office building palace pagoda train station. It was the highest building in the world. Burj painting pantry park Khalifa, the architectural style of which is Deconstructu- person parking lot patio vism, was built between 6 January 2004 and 31 December pavilion pier playground 2009. It was built of glass, steel, aluminium, reinforced con- pool/inside pub indoor pyramid restaurant restaurant kitchen restaurant patio crete. It was formerly called Burj Dubai. River rock arch rope bridge Ruin Schoolhouse sculpture Table 9: Burj Khalifa (eval set) Shed Shopfront shopping mall indoor sill ski resort Skyscraper smokestack Stable stained-glass Wikipedia (human) staircase structure–bridge structure–building Christ the Redeemer is an Art Deco statue of Jesus Christ structure–tunnel swimming pool indoor swimming pool outdoor in Rio de Janeiro, Brazil, created by French sculptor Paul synagogue outdoor temple asia throne room Landowski and built by Brazilian engineer Heitor da Silva Tower tower-pisa train station platform Costa, in collaboration with French engineer Albert Caquot. tree house Village water tower Romanian sculptor Gheorghe Leonida fashioned the face. waterfall wind farm windmill window youth hostel zen garden Constructed between 1922 and 1931, the statue is 30 metres (98 ft) high, excluding its 8-metre (26 ft) pedestal. The arms Table 8: List of classes supported by the object detec- stretch 28 metres (92 ft) wide. DBpedia tion module Christ the Redeemer (statue), which was built of Soapstone, A.3 Details on the visuals analysis is a Statue in Brazil. List of extracted visual features. Table8 shows Fused Christ the Redeemer (statue), which was built of Soapstone, is the list of all features extracted from images. a statue in a zen garden environment in Brazil. Its archi- Training of models. The training settings of each tectural style is Hellinistic. Christ the Redeemer (statue) has similarities with a windmill and a beach house. There component’s model involve a batch size of value is an elevator shaft in it. 2, learning rate of 0.0001, momentum value equal to 0.9, weight decay of 0.0005 and weights initial- Table 10: Christ the Redeemer (eval set)
13 Figure 7: Confusion matrix of the produced architectural styles model
Wikipedia (human) Wikipedia (human) The Dancing House, or Fred and Ginger, is the nickname The Sydney Opera House is a multi-venue performing arts given to the Nationale-Nederlanden building on the Rasˇ´ınovo centre at Sydney Harbour in Sydney, New South Wales, Aus- nab´ reˇ zˇ´ı (Rasˇ´ın Embankment) in Prague, Czech Republic. tralia. It is one of the 20th century’s most famous and distinc- It was designed by the Croatian-Czech architect Vlado tive buildings. Milunic´ in cooperation with Canadian-American architect DBpedia Frank Gehry on a vacant riverfront plot. The building was Sydney Opera House, which Jørn Utzon designed, is a Per- designed in 1992 and was completed four years later in 1996. forming arts center in Sydney. Sydney Opera House, the DBpedia architectural style of which is Expressionist architecture, was Dancing House, which Frank Gehry designed, is a Nickname built between 1 March 1959 and 1973. Its structure is made in CzechRepublic (Prague). It was built between 1992 and of Concrete frame & precast concrete ribbed roof. 1996. It was formerly called Fred and Ginger. Fused Fused Sydney Opera House, which Jørn Utzon designed, is a centre Dancing House, which Frank Gehry designed, is a nickname in Sydney. Its structure is made of Concrete frame & pre- in CzechRepublic (Prague). It has similarities with an em- cast concrete ribbed roof. An element of the structure is bassy, a palace and a parking garage. A fire escape can like a bridge. Sydney Opera House has similarities with a be seen on its facade. Dancing House, the architectural style beach house, an amusement park and a museum. Sydney of which is Art Nouveau, was built between 1992 and 1996. Opera House, the architectural style of which is Deconstruc- It was formerly called Fred and Ginger. tuvism, was built between 1 March 1959 and 1973.
Table 11: Dancing house (eval set) Table 12: Sydney Opera house (eval set)
14