WYSIWYM – Integrated Visualization, Exploration and Authoring of Un-Structured and Semantic Content

Semantic Web 1 (2013) 1–14 1 IOS Press

WYSIWYM – Integrated Visualization, Exploration and Authoring of Un-structured and Semantic Content

Ali Khalili ∗, Sören Auer, University of Leipzig, Institute of Computer Science, AKSW Group, Augustusplatz 10, D-04009 Leipzig, Germany E-mail: {lastname}@informatik.uni-leipzig.de

Abstract. The Semantic Web and Linked Data gained traction in the last years. However, the majority of information still is contained in unstructured documents. This can also not be expected to change, since text, images and videos are the natural way how humans interact with information. Semantic structuring on the other hand enables the (semi-)automatic integration, repurposing, rearrangement of information. NLP technologies and formalisms for the integrated representation of unstructured and semantic content (such as RDFa and Microdata) aim at bridging this semantic gap. However, in order for humans to truly beneﬁt from this integration, we need ways to author, visualize and explore unstructured and semantic information in a holistic manner. In this paper, we present the WYSIWYM (What You See is What You Mean) concept, which addresses this issue and formalizes the binding between semantic representation models and UI elements for authoring, visualizing and exploration. With RDFaCE and Pharmer we present and evaluate two complementary showcases implementing the WYSIWYM concept for different application domains.

Keywords: Visualization, Authoring, Exploration, Semantic Web, WYSIWYM, WYSIWYG, Visual Mapping

1. Introduction such as faceted search [27] or question answer- ing [16]. The Semantic Web and Linked Data gained traction – In information presentation semantically enriched in the last years. However, the majority of informa- documents can be used to create more sophis- tion still is contained in and exchanged using unstruc- ticated ways of flexibly visualizing information, tured documents, such as Web pages, text documents, such as by means of semantic overlays as de- images and videos. This can also not be expected to scribed in [3]. change, since text, images and videos are the natural – For information integration semantically enriched way how humans interact with information. Semantic documents can be used to provide unified views structuring on the other hand provides a wide range on heterogeneous data stored in different applica- of advantages compared to unstructured information. tions by creating composite applications such as It facilitates a number of important aspects of informa- semantic mashups [2]. tion management: – To realize personalization, semantic documents provide customized and context-specific informa- – For search and retrieval enriching documents tion which better fits user needs and will result with semantic representations helps to create in delivering customized applications such as per- more efficient and effective search interfaces, sonalized semantic portals [23]. – For reusability and interoperability enriching *Corresponding author. E-mail: [email protected] documents with semantic representations facili- leipzig.de. tates exchanging content between disparate sys-

tems and enables building applications such as knowledge representation formalisms as well as executable papers [18]. UI elements for authoring, visualization and exploration of such elements. 3. Two complementary use cases, which evaluate Natural Language Processing (NLP) technologies different, concrete WYSIWYM interfaces in a (e.g. named entity recognition and relationship extrac- generic as well as domain specific context. tion) as well as formalisms for the integrated representation of unstructured and semantic content (such as RDFa and Microdata) aim at bridging the semantic gap The WYSIWYM formalization can be used as a ba- between unstructured and semantic representation for- sis for implementations; allows to evaluate and classify malisms. However, in order for humans to truly benefit existing user interfaces in a defined way; provides a from this integration, we need ways to author, visual- terminology for software engineers, user interface and ize and explore unstructured and semantic information domain experts to communicate efficiently and effec- in a holistic manner. tively. We aim to contribute with this work to mak- In this paper, we present the WYSIWYM (What ing Semantic Web applications more user friendly and You See Is What You Mean) concept, which addresses ultimately to create an ecosystem of flexible UI com- the issue of an integrated visualization, exploration ponents, which can be reused, repurposed and chore- and authoring of un-structured and semantic content. ographed to accommodate the UI needs of dynami- Our WYSIWYM concept formalizes the binding be- cally evolving information structures. tween semantic representation models and UI elements The remainder of this article is structured as follows: for authoring, visualizing and exploration. We anal- In Section 2, we describe the background of our work yse popular tree, graph and hyper-graph based seman- and discuss the related work. Section 3 describes the tic representation models and elicit a list of semantic fundamental WYSIWYM concept proposed in the pa- representation elements, such as entities, various rela- per. Subsections of Section 3 present the different com- tionships and attributes. We provide a comprehensive ponents of the WYSIWYM model. In Section 4, we survey of common UI elements for authoring, visu- introduce two implemented WYSIWYM interfaces to- alizing and exploration, which can be configured and gether with their evaluation results. Finally, Section 5 bound to individual semantic representation elements. concludes with an outlook on future work. Our WYSIWYM concept also comprises cross-cutting helper components, which can be employed within a concrete WYSIWYM interface for the purpose of 2. Related Work automation, annotation, recommendation, personalization etc. WYSIWYG. The term WYSIWYG as an acronym for With RDFaCE and Pharmer we present and evalu- What-You-See-Is-What-You-Get is used in computing ate two complementary showcases implementing the to describe a system in which content (text and graph- WYSIWYM concept for different domains. RDFaCE ics) displayed on-screen during editing appears in a is domain agnostic editor for text content with em- form closely corresponding to its appearance when bedded semantic in the form of RDFa or Microdata. printed or displayed as a finished product. The first Pharmer is a WYSIWYM interface for the authoring of usage of the term goes back to 1974 in the print in- semantic prescriptions and thus targeting the medical dustry to express the idea that what the user sees on domain. Our evaluation of both tools with end-users the screen is what the user gets on the printer. Xe- (in case of RDFaCE) and domain experts (in case of rox PARC’s Bravo was the first WYSIWYG editor- Pharmer) shows, that WYSIWYM interfaces provide formatter [19]. It was designed by Butler Lampson and good usability, while retaining benefits of a truly se- Charles Simonyi who had started working on these mantic representation. concepts around 1970 while at Berkeley. Later on by The contributions of this work are in particular: the emergence of Web and HTML technology, the 1. A formalization of the WYSIWYM concept WYSIWYG concept was also utilized in Web-based based on definitions for the WYSIWYM model, text editors. The aim was to reduce the effort required binding and concrete interfaces. by users to express the formatting directly as valid 2. A comprehensive survey of semantic represen- HTML markup. In a WYSIWYG editor users can edit tation elements of tree, graph and hyper-graph content in a view which matches the final appear- Khalili et al. / WYSIWYM 3 ance of published content with respect to fonts, head- cept is to enrich the existing WYSIWYG presenta- ings, layout, lists, tables, images and structure. Be- tional view of the content with UI components reveal- cause using a WYSIWYG editor may not require any ing the semantics embedded in the content and enable HTML knowledge, they are often easier for an aver- the exploration and authoring of semantic content. In- age computer user to get started with. The first pro- stead of separating presentation, content and meaning, grams for building Web pages with a WYSIWYG in- our WYSIWYM approach aims to integrate these as- terface were Netscape Gold, Claris HomePage, and pects to facilitate the process of Semantic Content Au- Adobe PageMill. thoring. There are already some approaches (i.e. vi- WYSIWYG text authoring is meanwhile ubiquitous sual mapping techniques), which go into the direction on the Web and part of most content creation and of integrated visualization and authoring of structured management workflows. It is part of content manage- content. ment cystems (CMS), weblogs, wikis, fora, product Visual Mapping Techniques. Visual mapping tech- data management systems and online shops, just to niques (a.k.a. knowledge representation techniques) mention a few. However, the WYSIWYG model has are methods to graphically represent knowledge struc- been criticized, primarily for the verbosity, poor sup- tures. Most of them have been developed as paper- port of semantics and low quality of the generated based techniques for brainstorming, learning facilita- code and there have been voices advocating a change tion, outlining or to elicit knowledge structures. Ac- towards a WYSIWYM (What-You-See-Is-What-You- cording to their basic topology, most of them can be re- Mean) model [26,24]. lated to the following fundamentally different primary WYSIWYM. The first use of the WYSIWYM term approaches [6,25]: occurred in 1995 aiming to capture the separation of – Mind-Maps. Mind-maps are created by drawing presentation and content when writing a document. one central topic in the middle together with la- 1 The LyX editor was the first WYSIWYM word pro- beled branches and sub-branches emerging from cessor for structure-based content authoring. Instead it. Instead of distinct nodes and links, mind-maps of focusing on the format or presentation of the doc- only have labeled branches. A mind-map is a con- ument, a WYSIWYM editor preserves the intended nected directed acyclic graph with hierarchy as meaning of each element. For example, page head- its only type of relation. Outlines are a similar ers, sections, paragraphs, etc. are labeled as such in technique to show hierarchical relationships us- the editing program, and displayed appropriately in the ing tree structure. Mind-maps and outlines are not browser. Another usage of the WYSIWYM term was suitable for relational structures because they are by Power et al. [22] in 1998 as a solution for Sym- constrained to the hierarchical model. bolic Authoring. In symbolic authoring the author gen- – Concept Maps. Concept maps consist of labeled erates language-neutral “symbolic" representations of nodes and labeled edges linking all nodes to a the content of a document, from which documents in connected directed graph. The basic node and link each target language are generated automatically, us- structure of a connected directed labeled graph ing Natural Language Generation technology. In this also forms the basis of many other modeling ap- What-You-See-Is-What-You-Meant approach, the lan- proaches like Entity-Relationship (ER) diagrams guage generator was used to drive the user interface and Semantic Networks. These forms have the (UI) with support of localization and multilinguality. same basic structure as concept maps but with Using the WYSIWYM natural language generation more formal types of nodes and links. approach, the system generates a feed-back text for the – Spatial Hypertext. A spatial hypertext is a set user that is based on a semantic representation. This of text nodes that are not explicitly connected representation can be edited directly by the user by ma- but implicitly related through their spatial layout, nipulating the feedback text. e.g., through closeness and adjacency — similar The WYSIWYM term as defined and used in this to a pin-board. Spatial hypertext can show fuzzily paper targets the novel aspect of integrated visualiza- related items. To fuzzily relate two items in a spa- tion, exploration and authoring of unstructured and se- tial hypertext schema, they are simply placed near mantic content. The rationale of our WYSIWYM con- to each other, but possibly not quite as near as to a third object. This allows for so-called “construc- 1http://www.lyx.org/ tive ambiguity” and is an intuitive way to deal 4 Khalili et al. / WYSIWYM

with vague relations and orders. Spatial Hyper- WYSIWYM Interface text abandons the concept of explicitly interrelat- Helper components ing objects. Instead, it uses spatial positioning as Visualization Exploration Authoring the basic structure. Techniques Techniques Techniques Binding data to UI elements. There are already many Configs Configs Configs approaches and tools which address the binding between data and UI elements for visualizing and explor- Bindings ing semantically structured data. Dadzie and Rowe [4] Semantic Representation Data Models present the most exhaustive and comprehensive survey to date of these approaches. For example, Fres- nel [21] is a display vocabulary for core RDF concepts. Fig. 1. Schematic view of the WYSIWYM model. Fresnel’s two foundational concepts are lenses and formats. Lenses define which properties of an RDF re- source, or group of related resources, are displayed and as a basis for design and implementation of novel ap- how those properties are ordered. Formats determine plications for authoring, visualization, and exploration how resources and properties are rendered and provide of semantic content (cf. Section 4). The formalization hooks to existing styling languages such as CSS. serves the purpose of providing a terminology for soft- Parallax, Tabulator, Explorator, Rhizomer, Sgvizler, ware engineers, user interface and domain experts to Fenfire, RDF-Gravity, IsaViz and i-Disc for Topic communicate efficiently and effectively. It provides in- Maps are examples of tools available for visualizing sights into and an understanding of the requirements and exploring semantically structured data. In these as well as corresponding UI solutions for proper de- tools the binding between semantics and UI elements sign and implementation of semantic content manage- is mostly performed implicitly, which limits their ver- ment applications. Secondly, it allows to evaluate and satility. However, an explicit binding as advocated by classify existing user interfaces according to the con- our WYSIWYM model can be potentially added to ceptual model in a defined way. This will highlight some of these tools. the gaps in existing applications dealing with semantic In contrast to the structured content, there are many content. approaches and tools which allow binding semantic Figure 1 provides a schematic overview of the data to UI elements within unstructured content (cf. WYSIWYM concept. The rationale is that elements of our comprehensive literature study [10]). As an exam- a knowledge representation formalism (or data model) ple, Dido [9] is a data-interactive document which lets are connected to suitable UI elements for visualiza- end users author semantic content mixed with unstruc- tion, exploration and authoring. Formalizing this con- tured content in a web-page. Dido inherits data explo- ceptual model results in three core definitions (1) for ration capabilities from the underlying Exhibit2 frame- the abstract WYSIWYM model, (2) bindings between work. Loomp as a prove-of-concept for the One Click UI and representation elements as well as (3) a con- Annotation [1] strategy is another example in this concrete instantiation of the abstract WYSIWYM model, text. Loomp is a WYSIWYG web editor for enrich- which we call a WYSIWYM interface. ing content with RDFa annotations. It employs a partial mapping between UI elements and data to hide the Definition 1 (WYSIWYM model). The WYSIWYM complexity of creating semantic data. model can be formally defined as a quintuple (D,V,X,T,H) where: – D is a set of semantic representation data models, 3. WYSIWYM Concept where each Di ∈ D has an associated set of data

model elements EDi ; In this section we introduce the fundamental WYSI- – V is a set of tuples (v, Cv), where v is a visual- WYM concept and formalize key elements of the con- ization technique and Cv a set of possible conﬁg- cept. Formalizing the WYSIWYM concept has a num- urations for the visualization technique v; ber of advantages: First, the formalization can be used – X is a set of tuples (x, Cx), where x is an exploration technique and Cx a set of possible conﬁg- 2http://simile-widgets.org/exhibit/ urations for the exploration technique x; Khalili et al. / WYSIWYM 5

– T is a set of tuples (t, Ct), where t is an authoring high technique and Ct a set of possible configurations for the authoring technique t; Spatial hypertext – H is a set of helper components. Concept maps The WYSIWYM model represents an abstract concept from which concrete interfaces can be derived Hypergraph-based by means of bindings between semantic representation Mind-maps Topic Maps Graph-based XTM model elements and configurations of particular UI el- RDF LTM RDF/XML CTM ements. Turtle/N3/N-Triples AsTMa RDFa Complexity of visual mapping of visual Complexity Tree-based JSON-LD Definition 2 (Binding). A binding b is a function which Microdata

Microformat maps each element of a semantic representation model low low high e (e ∈ EDi ) to a set of tuples (ui, c), where ui is a Semantic expressiveness user interface technique ui (ui ∈ V ∪ X ∪ T ) and c is a configuration c ∈ Cui. Fig. 2. Comparison of existing visual mapping techniques in terms Figure 4 gives an overview on all data model of semantic expressiveness and complexity of visual mapping. (columns) and UI elements (rows) and how they can be bound together using a certain configuration (cells). of this sections we discuss the different parts of the The shades of gray in a certain cell indicate the suit- WYSIWYM concept in more detail. ability of a certain binding between a particular UI and data model element. Once a selection of data models 3.1. Semantic Representation Models and UI elements was made and both are bound to each other encoding a certain configuration in a binding, Semantic representation models are conceptual data we attain a concrete instantiation of our WYSIWYM models to express the meaning of information thereby model called WYSIWYM interface. enabling representation and interchange of knowledge. Definition 3 (WYSIWYM interface). An instantiation Based on their expressiveness, we can roughly divide of the WYSIWYM model I called WYSIWYM interface popular semantic representation models into the three now is a hextuple (DI ,VI ,XI ,TI ,HI , bI ), where: categories tree-based, graph-based and hypergraph- based (cf. Figure 2). Each semantic representation D is a selection of semantic representation data – I model comprises a number of representation elements, models (D ⊂ D); I such as various types of entities and relationships. – V is a selection of visualization techniques I For visualization, exploration and authoring it is of (V ⊂ V ); I paramount importance to bind the most suitable UI el- – X is a selection of exploration techniques I ements to respective representation elements. In the se- (X ⊂ X); I quel we briefly discuss the three different types of rep- – T is a selection of authoring techniques I resentation models. (TI ⊂ T ); – HI is a selection of helper components Tree-based. This is the simplest semantic represen- (HI ⊂ H); tation model, where semantics is encoded in a tree- – bI is a binding which binds a particular occur- like structure. It is suited for representing taxonomic rence of a data model element to a visualization, knowledge, such as thesauri, classification schemes, exploration and/or authoring technique. subject heading lists, concept hierarchies or mind- Note, that we limit the definition to one binding, maps. It is used extensively in biology and life sci- which means that only one semantic representation ences, for example, in the APG III system (Angiosperm model is supported in a particular WYSIWYM inter- Phylogeny Group III system) of flowering plant classi- face at a time. It could be also possible to support sev- fication, as part of the Dimensions of the XBRL (eXten- eral semantic representation models (e.g. RDFa and sible Business Reporting Language) or generically in Microdata) at the same time. However, this can be con- the SKOS (Simple Knowledge Organization System). fusing to the user, which is why we deliberately ex- Elements of tree-based semantic representation usu- cluded this case in our definition. In the remainder ally include: 6 Khalili et al. / WYSIWYM

– E1: Item – e.g. Magnoliidae, the item repre- RDF-based data can be serialized in various for- senting all ﬂowering plants. mats, such as RDFa, RDF/XML, JSON-LD or Turtle/N3/N- – E2: Item type – e.g. biological term for Triples. Magnoliidae. Hypergraph-based. A hypergraph is a generalization – E : Item-subitem relationships – e.g. Magnoliidae 3 of a graph in which an edge can connect any num- referring to subitem magnolias. ber of vertices. Since hypergraph-based models al- – E : Item property value – e.g. the synonym 4 low n-ary relationships between arbitrary number of flowering plant for the item Magnoliidae. nodes, they provide a higher level of expressiveness – E : Related items – e.g. the sibling item Eudicots 5 compared to tree-based and graph-based models. The to Magnoliidae. most prominent representative is the Topic Maps data Tree-based data can be serialized as Microdata or model developed as an ISO/IEC standard which con- Microformats. sists of topics, associations and occurrences. The semantic expressivity of Topic Maps is, in many ways, Graph-based. This semantic representation model equivalent to that of RDF, but the major differences are adds more expressiveness compared to simple tree- that Topic Maps (i) provide a higher level of seman- based formalisms. The most prominent representative tic abstraction (providing a template of topics, associa- is the RDF data model, which can be seen as a set of tions and occurrences, while RDF only provides a tem- triples consisting of subject, predicate, object, where plate of two arguments linked by one relationship) and each component can be a URI, the object can be a (hence) (ii) allow n-ary relationships (hypergraphs) be- literal and subject as well as object can be a blank tween any number of nodes, while RDF is limited to node. The most distinguishing features of RDF from triplets. The hypergraph-based model is suited for rep- a simple tree-based model are: the distinction of enti- resenting complex schemes such as spatial hypertext. ties in classes and instances as well as the possibility Hypergraph-based models are used for a variety of ap- to express arbitrary relationships between entities. The plications. Amongst them are musicDNA3 as an index graph-based model is suited for representing combi- of musicians, composers, performers, bands, artists, natorial schemes such as concept maps. Graph-based producers, their music, and the events that link them models are used in a very broad range of domains, together, TM4L (Topic Maps for e-Learning), clini- for example, in the FOAF (Friend of a Friend) for de- cal decision support systems and enterprise informa- scribing people, their interests and interconnections in tion integration. Elements of Topic Maps as a typical a social network, in MusicBrainz to publish informa- hypergraph-based data model are: tion about music albums, in the medical domain (e.g. DrugBank, Diseasome, ChEMBL, SIDER) to describe – E1: Topic name – e.g. University of Leipzig. the relations between diseases, drugs and genes, or – E2: Topic type – e.g. organization for Uni- generically in the SIOC (Semantically-Interlinked On- versity of Leipzig. line Communities) vocabulary. Elements of RDF as a – E3: Topic associations – e.g. member of a typical graph-based data model are: project which has other organization partners. – E4: Topic role in association e.g. coordinator. – E Warfarin 1: Instances – e.g. as a drug. – E5: Topic occurrences – e.g. address. – E2: Classes – e.g. anticoagulants drug Augustusplatz 10, for Warfarin. ∗ E51 : value – e.g. 04109 Leipzig. – E3: Relationships between entities (instances text or classes) – e.g. the interaction between ∗ E52 : datatype – e.g. . Aspirin as an antiplatelet drug and Warfarin Topic Maps-based data can be serialized as an which will increase the risk of bleeding. XML-based syntax called XTM (XML Topic Map), – E4: Literal property values – e.g. the halﬂife for LTM (Linear Topic Map Notation), CTM (Compact the Amoxicillin. Topic Maps Notation) and AsTMa (Asymptotic Topic Map Notation). ∗ E41 : Value – e.g. 61.3 minutes.

∗ E42 : Language tag – e.g. en. 3 ∗ E43 : Datatype – e.g. xsd:float. http://www.musicdna.info/ Khalili et al. / WYSIWYM 7

Fig. 3. Screenshots of user interface techniques for visualization and exploration: 1-framing using borders, 2-framing using backgrounds, 3-video subtitle, 4-line connectors and arrow connectors, 5-bar layouts, 6-text formatting, 7-image color effects, framing and line connectors, 8-expandable callout, 9-marking with icons, 10-tooltip callout, 11-faceting

3.2. Visualization types), background styles (colours, patterns) or overlay styles (when applied to images and videos) The primary objectives of visualization are to present, can be used to distinguish different types of se- transform, and convert semantic data into a visual rep- mantic entities (cf. Figure 3 no. 1, 2). The tech- resentation, so that, humans can read, query and edit nique is already employed in social networking them efﬁciently. We divide existing techniques for vi- websites such as Google Plus and Facebook to sualization of knowledge encoded in text, images and tag people within images. videos into the three categories Highlighting, Associ- – V2: Text formatting (color, font, size, etc.). In this ating and Detail view. Highlighting includes UI tech- technique different text styles such as font fam- niques which are used to distinguish or highlight a part ily, style, weight, size, colour, shadows and other of an object (i.e. text, image or video) from the whole text decoration techniques are used to distinguish object. Associating deals with techniques that visual- semantic entities within a text (cf. Figure 3 no. ize the relation between some parts of an object. Detail 6). The problem with this technique is that in view includes techniques which reveal detailed infor- an HTML document, the applied semantic styles mation about a part of an object. For each of the above might overlap with existing styles in the docu- categories, the related UI techniques are as follows: ment and thereby add ambiguity to recognizing semantic entities. - Highlighting. – V3: Image color effects. This technique is simi- – V1: Framing and Segmentation (borders, over- lar to text formatting but applied to images and lays and backgrounds). This technique can be ap- videos. Different image color effects such as plied to text, images and videos, we enclose a se- brightness/contrast, shadows, glows, bevel/emboss mantic entity in a coloured border, background or are used to highlight semantic entities within an overlay. Different border styles (colours, width, image (cf. Figure 3 no. 7). This technique suf- 8 Khalili et al. / WYSIWYM

fers from the problem that the applied effects they do not appear on mobile devices (by hover), might overlap with the existing effects in the im- since there is no cursor. age thereby making it hard to distinguish the se- – V10: Video subtitles. Subtitles are textual versions mantic entities. of the dialog or commentary in videos. They are – V4: Marking (icons appended to text or image). In usually displayed at the bottom of the screen and this technique, which can be applied to text, im- are employed for written translation of a dialog ages and videos, we append an icon as a marker in a foreign language. Video subtitles can be used to the part of object which includes the semantic to reflect detailed semantics embedded in a video entity (cf. Figure 3 no. 9). The most popular use scene when watching the video. A problem with of this technique is currently within maps to indi- subtitles is efficiently scaling the text size and re- cate specific points of interest. Different types of lating text to semantic entities when several se- icons can be used to distinguish different types of mantic entities exist in a scene. semantic or correlated entities. – V5: Bleeping. A bleep is a single short high- 3.3. Exploration pitched signal in videos. Bleeping can be used to highlight semantic entities within a video. Differ- To increase the effectiveness of visualizations, users ent type of bleep signals can be defined to distin- need to be capable to dynamically explore the visual guish different types of semantic entities. representation of the semantic data. The dynamic ex- – V6: Speech (in videos). In this technique a video ploration of semantic data will result in faster and eas- is augmented by some speech indicating the se- ier comprehension of the targeted content. Techniques mantic entities and their types within the video. for exploration of semantics encoded in text, images - Associating. and videos include: – X : Zooming. In a zoomable UI, users can change – V7: Line connectors. Using line connectors is the 1 simplest way to visualize the relation between se- the scale of the viewed area in order to see more mantic entities in text, images and videos (cf. Fig- detail or less. Zooming in a semantic entity can ure 3 no. 4). If the value of a property is avail- reveal further details such as property value or enable in the text, line connectors can also reflect tity type. Zooming out can be employed to reveal the item property values. Problematic is that nor- the relations between semantic entities in a text, mal line connectors can not express the direction image or video. Supporting rich dynamics by con- of a relation. figuring different visual representations for se- – V8: Arrow connectors. Arrow connectors are ex- mantic objects at different sizes is a requirement tended line connectors with arrows to express the for a zoomable UI. The iMapping approach[6] direction of a relation in a directed graph. which is implemented in the semantic desktop is an example of the zooming technique. - Detail view. – X2: Faceting. Faceted browsing is a technique – V9: Callouts. A callout is a string of text con- for accessing information organized according to nected by a line, arrow, or similar graphic to a part a faceted classification system, allowing users to of text, image or video giving information about explore a collection of information by applying that part. It is used in conjunction with a cursor, multiple filters (cf. Figure 3 no. 11). Defining usually a pointer. The user hovers the pointer over facets for each component of the predefined se- an item, without clicking it, and a callout appears mantic models enable users to browse the under- (cf. Figure 3 no. 10). Callouts come in different lying knowledge space by iteratively narrowing styles and templates such as infotips, tooltips, hint the scope of their quest in a predetermined order. and popups. Different sort of semantic informa- One of the main problems with faceted browsers tion can be embedded in a callout to indicate the is the increased number of choices presented to type of semantic entities, property values and re- the user at each step of the exploration [5]. lationships. Another variant of callouts is the sta- – X3: Bar layouts. In the bar layout, each semantic tus bar which displays semantic information in entity within the text is indicated by a vertical bar a bar appended to the text, image or video con- in the left or right margin (cf. Figure 3 no. 5). The tainer. A problem with dynamic callouts is that colour of the bar reflects the type of the entity. The Khalili et al. / WYSIWYM 9

bars are ordered by length and order in the text. ited set of choices that are available in the current Nested bars can be used to show the hierarchies state, or context. of entities. Semantic entities in the text are high- – T6: (Floating) Ribbon editing. A ribbon is a com- lighted by a mouse-over the corresponding bar. mand bar that organizes functions into a series of This approach is employed in Loomp [17]. tabs or toolbars at the top of the editable content. – X4: Expandable callouts. Expandable callouts Ribbon tabs/toolbars are composed of groups, are interactive and dynamic callouts which enable which are a labeled set of closely related com- users to explore the semantic data associated to mands. A floating ribbon is a ribbon that appears a predefined semantic entity (cf. Figure 3 no. 8). when user rolls the mouse over a target area. This technique is employed in OntosFeeder [11]. A floating ribbon increases usability by bringing edit functions as close as possible to the user’s 3.4. Authoring point of focus. The Aloha WYSIWYG editor4 is an example of floating ribbon based content author- Semantic authoring aims to add more meaning to ing. digitally published documents. If users do not only – T7: Voice commands. Voice commands permit the publish the content, but at the same time describe what user’s hands and eyes to be busy with another it is they are publishing, then they have to adopt a task, which is particularly valuable when users structured approach to authoring. A semantic author- are in motion or outside. Users tend to prefer ing UI is a human accessible interface with capabil- speech for functions like describing objects, sets ities for writing and modifying semantic documents. and subsets of objects [20]. The following techniques can be used for authoring of – T8: (Multi-touch) gestures. A gesture (a.k.a. sign semantics encoded in text, images and videos: language) is a form of non-verbal communica- tion in which visible bodily actions communicate – T1: Form editing. In form editing, a user employs particular messages. Technically, different meth- existing form elements such as input/check/radio ods can be used for detecting and identifying ges- boxes, drop-down menu, slider, spinner, buttons, tures. Movement-sensor-based and camera-based date/color picker etc. for content authoring. approaches are two commonly used methods for – T2: Inline edit. Inline editing is the process of the recognition of in-air gestures [15]. Multi- editing items directly in the view by performing touch gestures are another type of gestures which simple clicks, rather than selecting items and then are defined to interact with multi-touch devices navigating to an edit form and submitting changes such as modern smartphones and tablets. Users from there. can use gestures to determine semantic entities, – T3: Drawing. Drawing as part of informal user their types and relationship among them. The interfaces [14], provides a natural human input main problem with gestures is their high level of to annotate an object by augmenting the ob- abstraction which makes it hard to assert concrete ject with human-understandable sketches. For in- property values. stance, users can draw a frame around semantic entities, draw a line between related entities etc. 3.5. Bindings Special shapes can be drawn to indicate different entity types or entity roles in a relation. Figure 4 surveys possible bindings between the user – T4: Drag and drop. Drag and drop is a pointing interface and semantic representation elements. The device gesture in which the user selects a virtual bindings were derived based on the following method- object by grabbing it and dragging it to a different ology: location or onto another virtual object. In general, it can be used to invoke many kinds of actions, or 1. We first analyzed existing semantic representa- create various types of associations between two tion models and extracted the corresponding ele- abstract objects. ments for each semantic model. 2. We performed an extensive literature study re- – T5: Context menu. A context menu (also called contextual, shortcut, or pop-up menu) is a menu garding existing approaches for visual mapping that appears upon user interaction, such as a right button mouse click. A context menu offers a lim- 4http://aloha-editor.org 10 Khalili et al. / WYSIWYM

as well as approaches addressing the binding be- employ existing NLP services (e.g. named en- tween data and UI elements. If the approach was tity recognition, relationship extraction) for auto- explicitly mentioning the binding composed of matic text annotation. UI elements and semantic model elements, we – H2: Real-time tagging is an extension of automa- added the binding to our mapping table. tion, which allows to create annotations proac- 3. We analyzed existing tools and applications tively while the user is authoring a text, image or which were implicitly addressing the binding be- video. This will significantly increase the annota- tween data and UI elements. tion speed and users are not distracted since they 4. Finally, we followed a predictive approach. We do not have to interrupt their current authoring investigated additional UI elements which are task. listed in existing HCI glossaries and carefully an- – H3: Recommendation means providing users alyzed their potential to be connected to a seman- with pre-filled form fields, suggestions (e.g. for tic model element. URIs, namespaces, properties), default values etc. These facilities simplify the authoring process, Although we deem the bindings to be fairly complete, as they reduce the number of required user inter- new UI elements might be developed or additional data actions. Moreover, they help preventing incom- models (or variations of the ones considered) might ap- plete or empty metadata. In order to leverage pear, in this case the bindings can be easily extended. other user’s annotations as recommendations, ap- The following binding configurations are available proaches like Paragraph Fingerprinting [8] can and refereed to from the cells of Figure 4: be implemented. – Defining a special border or background style – H4: Personalization and context-awareness de- (C1), text style (C2), image color effect (C4), scribes the ability of the UI to be configured ac- beep sound (C5), bar style (C6), sketch (C7), cording to users’ contexts, background knowl- draggable or droppable shape (C8), voice com- edge and preferences. Instead of being static, a mand (C9), gesture (C10) or a related icon (C3) personalized UI dynamically tailors its visual- for each type. ization, exploration and authoring functionalities based on the user profile and context. – Progressive shading (C11) by defining continuous shades within a specific color scheme to distin- – H5: Collaboration and crowdsourcing enables guish items in different levels of the hierarchy. collaborative semantic authoring, where the authoring process can be shared among differ- – Hierarchical bars (C12) by defining special styles for nested bars. ent authors at different locations. There are a – Grouping by similar border or background style vast amounts of amateur and expert users which are collaborating and contributing on the Social (C13), text style (C14), icons (C15) or image color Web. Crowdsourcing harnesses the power of such effects (C16). crowds to significantly enhance and widen the 3.6. Helper Components results of semantic content authoring and annotation. Generic approaches for exploiting single- In order to facilitate, enhance and customize the user Web applications for shared editing [7] can WYSIWYM model, we utilize a set of helper com- be employed in this context. H Accessibility ponents, which implement cross-cutting aspects. A – 6: means providing people with helper component acts as an extension on top of the disabilities and special needs with appropriate core functionality of the WYSIWYM model. The fol- UIs. The underlying semantic model in a WYSI- WYM UI can allow alternatives or conditional lowing components can be used to improve the quality content in different modalities to be selected of a WYSIWYM UI depending on the requirements based on the type of the user disability and infor- defined for a specific application domain: mation need. – H1: Automation means the provision of facil- – H7: Multilinguality means supporting multiple ities for automatic annotation of text, images languages in a WYSIWYM UI when visualizing, and videos to reduce the need for human work exploring or authoring the content. and thereby facilitating the efficient annotation of large item collections. For example, users can Khalili et al. / WYSIWYM 11

* If value is available in the text/subtitle. Tree-based Graph-based Hypergraph-based (e.g. Taxonomies) (e.g. RDF) (e.g. Topic Maps) No binding Partial binding Full binding

Literal property Topic

Occurre

values nces

type

association

subitem

in in

Item -

Topic

Class

associations

entities Instance

type Item

Topic Topic role role

Item

Related Related Items

Value Value

Datatype Datatype

Topic Topic

Item property property Item value

Relationships Relationships between Language Language tag

Structure Topic UI categories UI techniques encoded in: Framing and segmentation (borders, C1 C11 C13 C1 C1 overlays, backgrounds) Highlighting Text formatting (color, C C C C 2 C font, size etc.) 2 11 14 2

text Marking (appended icons) C3 C15 C3 C3 Line connectors * * * Associating Arrow connectors * * * Callouts Detail view (infotips, tooltips, popups) Framing and segmentation (borders, C1 C11 C13 C1 C1 overlays, backgrounds) Highlighting C4 Image color effects C4 C11 C16 C4

images Marking (appended icons) C3 C15 C3 C3 Line connectors Associating Arrow connectors Callouts Visualization Detail view (infotips, tooltips, popups) Framing and segmentation (borders, C1 C11 C13 C1 C1 overlays, backgrounds)

Image color effects C4 C11 C16 C4 C4 Highlighting C C Marking (appended icons) C 15 3 C 3 3

videos Bleeping C5 C5 C5 Speech Line connectors * * * Associating Arrow connectors * * * Callouts

Detail view (infotips, tooltips, popups) Subtitle Zooming

Faceting text Bar layout C5 C12 C5 C5 Expandable callouts Zooming images Exploration Faceting videos Faceting (excerpts) Form editing Inline edit

Drawing C7 C7 C7 C7 Drag and drop C C C C text, images, 8 8 8 8 videos Context menu

Authoring (Floating) Ribbon editing

Voice commands C9 C9 C9 C9 (Multi-Touch) Gestures C10 C10 C10 C10

Fig. 4. Possible bindings between user interface and semantic representation model elements. 12 Khalili et al. / WYSIWYM

4. Implementation and Evaluation Usability Factor/Grade Poor Fair Neutral Good Excellent Fit for use 0% 12.50% 31.25% 43.75% 12.50% Ease of learning 0% 12.50% 50% 31.25% 6.25% In order to evaluate the WYSIWYM model, we im- Task efficiency 0% 0% 56.25% 37.50% 6.25% plemented the two applications RDFaCE and Pharmer, Ease of remembering 0% 0% 37.50% 50% 12.50% which we present in the sequel. Subjective satisfaction 0% 18.75% 50% 25% 6.25% Understandability 6.25% 18.75% 31.25% 37.50% 6.25% RDFaCE. RDFaCE (RDFa Content Editor) is a Table 1 WYSIWYM interface for semantic content author- Usability evaluation results for RDFaCE. ing. It is implemented on top of the TinyMCE rich text editor. RDFaCE extends the existing WYSIWYG user interfaces to facilitate semantic authoring within popular CMSs, such as blogs, wikis and discussion forums. The RDFaCE implementation (cf. Figure 5, left) is open-source and available for download together with an explanatory video and online demo at http://aksw.org/Projects/RDFaCE. RD- FaCE as a WYSIWYM instantiation can be described using the following hextuple: – D: RDFa, Microdata5. – V: Framing using borders (C: special border color defined for each type), Callouts using dynamic tooltips. Fig. 6. Usability evaluation results for Pharmer (0: Strongly dis- – E: Faceting based on the type of entities. agree, 1: Disagree, 2: Neutral, 3: Agree, 4: Strongly agree). – T: Form editing, Context Menu, Ribbon editing. minutes was available to use different features of RD- – H: Automation, Recommendation. FaCE for annotating occurrences of persons, locations – b: bindings defined in Figure 4. and organizations with suitable entity references. Sub- RDFaCE comes with a special edition customized sequently, a survey was presented to the participants for Schema.org vacabularies6. In this version, dif- where they were asked questions about their experi- ferent color schemes are defined for vocabularies ence while working with RDFaCE. Questions were tar- at Schema.org. Users are able to create a subset of geting six factors of usability [12] namely Fit for use, Schema.org schemas for their intended domain and Ease of learning, Task efficiency, Ease of remember- customize the colors for this subset. In this version, ing, Subjective satisfaction and Understandability. Re- nested forms are dynamically generated from the se- sults of the survey are shown in Table 1. They indi- lected schemas for authoring and editing of the anno- cate on average good to excellent usability for RD- tations. FaCE. A majority of the users deem RDFaCE being fit In order to evaluate RDFaCE usability, we con- for use and its functionality easy to remember. Also, ducted an experiment with 16 participants of the ISS- easy of learning and subjective satisfaction was well LOD 2011 summer school7. The user evaluation com- rated by the participants. There was a slightly lower prised the following steps: First, some basic informa- (but still above average) assessment of task efficiency tion about semantic content authoring along with a and understandability, which we attribute to the short demo showcasing different RDFaCE features was pre- time participants had for familiarizing themselves with sented to the participants as a 3 minutes video. Then, RDFaCE and the quite comprehensive functionality, participants were asked to use RDFaCE to annotate which includes automatic annotations, recommenda- three text snippets – a wiki article, a blog post and a tions and various WYSIWYM UI elements. news article. For each text snippet, a timeslot of five Pharmer. Pharmer is a WYSIWYM interface for the authoring of semantically enriched electronic pre- 5 Microdata support is implemented in RDFaCE-Lite available at scriptions. It enables physicians to embed drug-related http://rdface.aksw.org/lite 6http://rdface.aksw.org/new metadata into e-prescriptions thereby reducing the 7Summer school on Linked Data: http://lod2.eu/ medical errors occurring in the prescriptions and in- Article/ISSLOD2011 creasing the awareness of the patients about the pre- Khalili et al. / WYSIWYM 13

Fig. 5. Screenshots of our two implemented WYSIWYM interfaces. Left: RDFaCE for semantic text authoring (T6 indicates the RDFaCE menu bar, V1 – the framing of named entities in the text, V9 – a callout showing additional type information, T5 – a context menu for revising annotations). Right: Pharmer for authoring of semantic prescriptions (V1 – highlighting of drugs through framing, V9 – additional information about a drug in a callout, T1/T2 combined form and inline editing of electronic prescriptions). scribed drugs and drug consumption in general. In range of 0 to 100 which represents a composite mea- contrast to database-oriented e-prescriptions, semantic sure of the overall usability of the system. The results prescriptions are easily exchangeable among other e- of our survey (cf. Figure 6) showed a mean usability health systems without need to changing their related score of 75 for Pharmer WYSIWYM interface which infrastructure. The Pharmer implementation (cf. Fig- indicates a good level of usability. Participants particu- ure 5, right) is open-source and available for down- larly liked the integration of functionality and the ease load together with an explanatory video and on- of learning and use. The confidence in using the sys- line demo8 at http://code.google.com/p/ tem was slightly lower, which we again attribute to the pharmer/. It is based on HTML5 contenteditable el- short learning phase and diverse functionality. ement. Pharmer as a WYSIWYM instantiation is defined using the following hextuple: – D: RDFa. 5. Conclusions – V: Framing using borders and background (C: special background color defined for each type), Bridging the gap between unstructured and seman- Callouts using dynamic popups. tic content is a crucial aspect for the ultimate suc- – E: Faceting based on the type of entities. cess of semantic technologies. With the WYSIWYM – T: Form editing, Inline edit. concept we presented in this article an approach for – H: Automation, Real-time tagging, Recommen- integrated visualization, exploration and authoring of dation. unstructured and semantic content. The WYSIWYM – b: bindings defined in Figure 4. model binds elements of a knowledge representation formalism (or data model) to a set of suitable UI el- In order to evaluate the usability of Pharmer, we per- ements for visualization, exploration and authoring. formed a user study with 13 subjects. Subjects were 3 Based on such a declarative binding mechanism, we physicians, 4 pharmacist, 3 pharmaceutical researchers aim to increase the flexibility, reusability and develop- and 3 students. We first showed them a 3-minute tuto- ment efficiency of semantics-rich user interfaces rial video of using different features of Pharmer then We deem this work as a first step in a larger research asked each one to create a semantic prescription with agenda aiming at improving the usability of seman- Pharmer. After finishing the task, we asked the partic- tic user interfaces, while retaining semantic richness ipants to fill out a questionnaire. We used the System and expressivity. In future work we envision to adopt a Usability Scale (SUS) [13] as a standardized, simple, model-driven approach to enable automatic implemen- ten item Likert scale-based questionnaire to grade the tation of WYSIWYM interfaces by user-defined pref- usability of Pharmer. SUS yields a single number in the erences. This will help to reuse, re-purpose and chore- ograph WYSIWYM UI elements to accommodate the 8http://bitili.com/pharmer needs of dynamically evolving information structures 14 Khalili et al. / WYSIWYM and ubiquitous interfaces. We also aim to bootstrap an [11] A. Klebeck, S. Hellmann, C. Ehrlich, and S. Auer. Ontosfeeder ecosystem of WYSIWYM instances and UI elements - a versatile semantic context provider for web content author- to support structure encoded in different modalities, ing. In ISWC 2011, volume 6644 of LNCS, pages 456–460. [12] S. Lauesen. User Interface Design: A Software Engineering such as images and videos. Creating live and context- Perspective. Addison Wesley, Feb. 2005. sensitive WYSIWYM interfaces which can be gener- [13] J. Lewis and J. Sauro. The Factor Structure of the System ated on-the-fly based on the ranking of available UI Usability Scale. In Human Centered Design, volume 5619 of elements is another promising research venue. LNCS, pages 94–103. 2009. [14] J. Lin, M. Thomsen, and J. A. Landay. A visual language for sketching large and complex interactive designs. CHI ’02, Acknowledgments pages 307–314. ACM. [15] A. Loecken, T. Hesselmann, M. Pielot, N. Henze, and S. Boll. User-centred process for the definition of free-hand gestures We would like to thank our colleagues from the applied to controlling music playback. Multimedia Syst., AKSW research group for their helpful comments and 18(1):15–31, 2012. inspiring discussions during the development of the [16] V. Lopez, V. Uren, M. Sabou, and E. Motta. Is question answer- WYSIWYM model. This work was partially supported ing fit for the semantic web? a survey. Semantic Web ? Inter- by a grant from the European Union’s 7th Framework operability, Usability, Applicability, 2(2):125–155, September 2011. Programme provided for the project LOD2 (GA no. [17] R. H. M. Luczak-Roesch. Linked data authoring for 257943). non-experts. In WWW WS on Linked Data on the Web (LDOW2009), 2009. [18] W. Muller, I. Rojas, A. Eberhart, P. Haase, and M. Schmidt. A- References r-e: The author-review-execute environment. Procedia Com- puter Science, 4:627 – 636, 2011. ICCS 2011. [1] One Click Annotation. volume 699 of CEUR Workshop Pro- [19] B. A. Myers. A brief history of human-computer interaction ceedings, February 2010. technology. interactions, 5(2):44–54, 1998. [2] A. Ankolekar, M. Krötzsch, T. Tran, and D. Vrandecic. The [20] S. Oviatt, P. Cohen, L. Wu, J. Vergo, L. Duncan, B. Suhm, two cultures: mashing up web 2.0 and the semantic web. In J. Bers, T. Holzman, T. Winograd, J. Landay, J. Larson, WWW 2007, pages 825–834. and D. Ferro. Designing the user interface for multimodal [3] G. Burel, A. E. Cano1, and V. Lanfranchi. Ozone browser: speech and pen-based gesture applications: state-of-the-art sys- Augmenting the web with semantic overlays. volume 449 of tems and future research directions. Hum.-Comput. Interact., CEUR WS Proceedings, June 2009. 15(4):263–322, Dec. 2000. [4] A.-S. Dadzie and M. Rowe. Approaches to visualising linked [21] E. Pietriga, C. Bizer, D. R. Karger, and R. Lee. Fresnel: A data: A survey. Semantic Web, 2(2):89–124, 2011. browser-independent presentation vocabulary for rdf. In ISWC, [5] L. Deligiannidis, K. J. Kochut, and A. P. Sheth. RDF data LNCS, pages 158–171. Springer, 2006. exploration and visualization. In CIMS 2007, pages 39–46. [22] R. Power, R. Power, D. Scott, D. Scott, R. Evans, and R. Evans. ACM, 2007. What you see is what you meant: direct knowledge editing with [6] H. Haller and A. Abecker. imapping: a zooming user interface natural language feedback, 1998. approach for personal and semantic knowledge management. [23] M. Sah, W. Hall, N. M. Gibbins, and D. C. D. Roure. Sem- SIGWEB Newsl., pages 4:1–4:10, Sept. 2010. port - a personalized semantic portal. In 18th ACM Conf. on [7] M. Heinrich, F. Lehmann, T. Springer, and M. Gaedke. Ex- Hypertext and Hypermedia, pages 31–32, 2007. ploiting single-user web applications for shared editing: a [24] C. Sauer. What you see is wiki – questioning WYSIWYG in generic transformation approach. In WWW 2012, pages 1057– the Internet age. In Proceedings of Wikimania 2006, 2006. 1066. ACM, 2012. [25] B. Shneiderman. Creating creativity: user interfaces for sup- [8] L. Hong and E. H. Chi. Annotate once, appear anywhere: col- porting innovation. ACM Trans. Comput.-Hum. Interact., lective foraging for snippets of interest using paragraph finger- 7(1):114–138, Mar. 2000. printing. CHI ’09, pages 1791–1794. ACM. [26] J. Spiesser and L. Kitchen. Optimization of html automatically [9] D. R. Karger, S. Ostler, and R. Lee. The web page as a wysi- generated by wysiwyg programs. In WWW 2004, pages 355– wyg end-user customizable database-backed information man- 364. agement application. In UIST 2009, pages 257–260. ACM, [27] D. Tunkelang. Faceted Search (Synthesis Lectures on Informa- 2009. tion Concepts, Retrieval, and Services). Morgan and Claypool [10] A. Khalili and S. Auer. User interfaces for semantic authoring Publishers, June 2009. of textual content: A systematic literature review. 2012.