WYSIWYM – Integrated Visualization, Exploration and Authoring of Un-Structured and Semantic Content

WYSIWYM – Integrated Visualization, Exploration and Authoring of Un-structured and Semantic Content

Ali Khalili Sören Auer Universität Leipzig, IFI/AKSW Universität Leipzig, IFI/AKSW PO 100920, D-04009 Leipzig PO 100920, D-04009 Leipzig [email protected] [email protected]

ABSTRACT information management: The Semantic Web and Linked Data gained traction in the • For search and retrieval enriching documents with se- last years. However, the majority of information still is con- mantic representations helps to create more efficient tained in unstructured documents. This can also not be and effective search interfaces, such as faceted search [27] expected to change, since text, images and videos are the or question answering [16]. natural way how humans interact with information. Se- • In information presentation semantically enriched doc- mantic structuring on the other hand enables the (semi- uments can be used to create more sophisticated ways )automatic integration, repurposing, rearrangement of in- of flexibly visualizing information, such as by means of formation. NLP technologies and formalisms for the inte- semantic overlays as described in [3]. grated representation of unstructured and semantic content • For information integration semantically enriched doc- (such as RDFa and Microdata) aim at bridging this seman- uments can be used to provide unified views on hetero- tic gap. However, in order for humans to truly benefit from geneous data stored in different applications by creat- this integration, we need ways to author, visualize and ex- ing composite applications such as semantic mashups [2]. plore unstructured and semantic information in a holistic • To realize personalization, semantic documents pro- manner. In this paper, we present the WYSIWYM (What vide customized and context-specific information which You See is What You Mean) concept, which addresses this better fits user needs and will result in delivering cus- issue and formalizes the binding between semantic repre- tomized applications such as personalized semantic por- sentation models and UI elements for authoring, visualizing tals [23]. and exploration. With RDFaCE and Pharmer we present • For reusability and interoperability enriching documents and evaluate two complementary showcases implementing with semantic representations facilitates exchanging the WYSIWYM concept for different application domains. content between disparate systems and enables building applications such as executable papers [18]. Categories and Subject Descriptors H.5.2 [User interfaces]: GUIs, Interaction styles Natural Language Processing (NLP) technologies (e.g. named entity recognition and relationship extraction) as well as formalisms for the integrated representation of unstructured General Terms and semantic content (such as RDFa and Microdata) aim Design, Human factors at bridging the semantic gap between unstructured and semantic representation formalisms. However, in order for hu- 1. INTRODUCTION mans to truly benefit from this integration, we need ways The Semantic Web and Linked Data gained traction in the to author, visualize and explore unstructured and semantic last years. However, the majority of information still is information in a holistic manner. contained in and exchanged using unstructured documents, such as Web pages, text documents, images and videos. This In this paper, we present the WYSIWYM (What You See can also not be expected to change, since text, images and Is What You Mean) concept, which addresses the issue of videos are the natural way how humans interact with infor- an integrated visualization, exploration and authoring of mation. Semantic structuring on the other hand provides un-structured and semantic content. Our WYSIWYM con- a wide range of advantages compared to unstructured in- cept formalizes the binding between semantic representation formation. It facilitates a number of important aspects of models and UI elements for authoring, visualizing and exploration. We analyse popular tree, graph and hyper-graph based semantic representation models and elicit a list of semantic representation elements, such as entities, various relationships and attributes. We provide a comprehensive survey of common UI elements for authoring, visualizing and exploration, which can be configured and bound to individ- ual semantic representation elements. Our WYSIWYM concept also comprises cross-cutting helper components, which can be employed within a concrete WYSIWYM interface for the purpose of automation, annotation, recommenda- as valid HTML markup. In a WYSIWYG editor users can tion, personalization etc. edit content in a view which matches the final appearance of published content with respect to fonts, headings, lay- With RDFaCE and Pharmer we present and evaluate two out, lists, tables, images and structure. The first programs complementary showcases implementing the WYSIWYM con- for building Web pages with a WYSIWYG interface were cept for different domains. RDFaCE is domain agnostic ed- Netscape Gold, Claris HomePage, and Adobe PageMill. itor for text content with embedded semantic in the form of RDFa or Microdata. Pharmer is a WYSIWYM interface for WYSIWYG text authoring is meanwhile ubiquitous on the the authoring of semantic prescriptions and thus targeting Web and part of most content creation and management the medical domain. Our evaluation of both tools with end- workflows. It is part of content management cystems (CMS), users (in case of RDFaCE) and domain experts (in case of weblogs, wikis, fora, product data management systems and Pharmer) shows, that WYSIWYM interfaces provide good online shops, just to mention a few. However, the WYSI- usability, while retaining benefits of a truly semantic repre- WYG model has been criticized, primarily for the verbosity, sentation. poor support of semantics and low quality of the generated code and there have been voices advocating a change The contributions of this work are in particular: towards a WYSIWYM (What-You-See-Is-What-You-Mean) 1. A formalization of the WYSIWYM concept based on model [26, 24]. definitions for the WYSIWYM model, binding and concrete interfaces. WYSIWYM. The first use of the WYSIWYM term occurred 2. A comprehensive survey of semantic representation el- in 1995 aiming to capture the separation of presentation and ements of tree, graph and hyper-graph knowledge rep- content when writing a document. The LyX editor 1 was the resentation formalisms as well as UI elements for au- first WYSIWYM word processor for structure-based content thoring, visualization and exploration of such elements. authoring. Instead of focusing on the format or presenta- 3. Two complementary use cases, which evaluate differ- tion of the document, a WYSIWYM editor preserves the in- ent, concrete WYSIWYM interfaces in a generic as tended meaning of each element. For example, page headers, well as domain specific context. sections, paragraphs, etc. are labeled as such in the editing program, and displayed appropriately in the browser. An- The WYSIWYM formalization can be used as a basis for im- other usage of the WYSIWYM term was by Power et al. [22] plementations; allows to evaluate and classify existing user in 1998 as a solution for Symbolic Authoring. In symbolic interfaces in a defined way; provides a terminology for soft- authoring the author generates language-neutral “symbolic” ware engineers, user interface and domain experts to com- representations of the content of a document, from which municate efficiently and effectively. We aim to contribute documents in each target language are generated automati- with this work to making Semantic Web applications more cally, using Natural Language Generation technology. user friendly and ultimately to create an ecosystem of flex- ible UI components, which can be reused, repurposed and The WYSIWYM term as defined and used in this paper tar- choreographed to accommodate the UI needs of dynamically gets the novel aspect of integrated visualization, exploration evolving information structures. and authoring of unstructured and semantic content. The rationale of our WYSIWYM concept is to enrich the ex- The remainder of this article is structured as follows: In isting WYSIWYG presentational view of the content with Section 2, we describe the background of our work and dis- UI components revealing the semantics embedded in the cuss the related work. Section 3 describes the fundamen- content and enable the exploration and authoring of seman- tal WYSIWYM concept proposed in the paper. Subsec- tic content. Instead of separating presentation, content and tions of Section 3 present the different components of the meaning, our WYSIWYM approach aims to integrate these WYSIWYM model. In Section 4, we introduce two imple- aspects to facilitate the process of Semantic Content Author- mented WYSIWYM interfaces together with their evalua- ing. There are already some approaches (i.e. visual mapping tion results. Finally, Section 5 concludes with an outlook on techniques), which go into the direction of integrated visu- future work. alization and authoring of structured content.

2. RELATED WORK Visual Mapping Techniques. Visual mapping techniques (a.k.a. knowledge representation techniques) are methods to WYSIWYG. The term WYSIWYG as an acronym for What- graphically represent knowledge structures. Most of them You-See-Is-What-You-Get is used in computing to describe have been developed as paper-based techniques for brain- a system in which content (text and graphics) displayed on- storming, learning facilitation, outlining or to elicit knowl- screen during editing appears in a form closely correspond- edge structures. According to their basic topology, most of ing to its appearance when printed or displayed as a fin- them can be related to the following fundamentally different ished product. The first usage of the term goes back to primary approaches [6, 25]: 1974 in the print industry to express the idea that what • Mind-Maps. Mind-maps are created by drawing one the user sees on the screen is what the user gets on the central topic in the middle together with labeled branches printer. Xerox PARC’s Bravo was the first WYSIWYG and sub-branches emerging from it. Instead of dis- editor-formatter [19]. Later on by the emergence of Web tinct nodes and links, mind-maps only have labeled and HTML technology, the WYSIWYG concept was also branches. A mind-map is a connected directed acyclic utilized in Web-based text editors. The aim was to reduce 1 the effort required by users to express the formatting directly http://www.lyx.org/ graph with hierarchy as its only type of relation. WYSIWYM Interface • Concept Maps. Concept maps consist of labeled nodes and labeled edges linking all nodes to a connected di- Helper components rected graph. The basic node and link structure of a Visualization Exploration Authoring connected directed labeled graph also forms the ba- Techniques Techniques Techniques sis of many other modeling approaches like Entity- Relationship (ER) diagrams and Semantic Networks. Configs Configs Configs These forms have the same basic structure as concept maps but with more formal types of nodes and links. Bindings • Spatial Hypertext. A spatial hypertext is a set of text nodes that are not explicitly connected but implic- Semantic Representation Data Models itly related through their spatial layout, e.g., through closeness and adjacency — similar to a pin-board. Spa- tial hypertext can show fuzzily related items. To fuzzily relate two items in a spatial hypertext schema, they Figure 1: Schematic view of the WYSIWYM model. are simply placed near to each other, but possibly not quite as near as to a third object. This allows for so- called “constructive ambiguity” and is an intuitive way classify existing user interfaces according to the conceptual to deal with vague relations and orders. model in a defined way. Also, the formalization serves the purpose of providing a terminology for software engineers, Binding data to UI elements. There are already many ap- user interface and domain experts to communicate efficiently proaches and tools which address the binding between data and effectively. and UI elements for visualizing and exploring semantically structured data. Dadzie and Rowe [4] present the most Figure 1 provides a schematic overview of the WYSIWYM exhaustive and comprehensive survey to date of these ap- concept. The rationale is that elements of a knowledge rep- proaches. For example, Fresnel [21] is a display vocabulary resentation formalism (or data model) are connected to suit- for core RDF concepts. Fresnel’s two foundational concepts able UI elements for visualization, exploration and author- are lenses and formats. Lenses define which properties of an ing. Formalizing this conceptual model results in three core RDF resource, or group of related resources, are displayed definitions (1) for the abstract WYSIWYM model, (2) bind- and how those properties are ordered. Formats determine ings between UI and representation elements as well as (3) how resources and properties are rendered and provide hooks a concrete instantiation of the abstract WYSIWYM model, to existing styling languages such as CSS. which we call a WYSIWYM interface. Parallax, Tabulator, Explorator, Rhizomer, Sgvizler, Fenfire, RDF-Gravity, IsaViz and i-Disc for Topic Maps are exam- ples of tools available for visualizing and exploring seman- Definition 1 (WYSIWYM model). The WYSIWYM tically structured data. In these tools the binding between model can be formally defined as a quintuple (D,V,X,T,H) semantics and UI elements is mostly performed implicitly, where: which limits their versatility. However, an explicit binding • D is a set of semantic representation data models, as advocated by our WYSIWYM model can be potentially where each Di ∈ D has an associated set of data model added to some of these tools. elements EDi ; • V is a set of tuples (v, Cv), where v is a visualization In contrast to the structured content, there are many ap- technique and Cv a set of possible configurations for proaches and tools which allow binding semantic data to UI the visualization technique v; elements within unstructured content (cf. our comprehen- • X is a set of tuples (x, Cx), where x is an exploration sive literature study [10]). As an example, Dido [9] is a data- technique and Cx a set of possible configurations for interactive document which lets end users author seman- the exploration technique x; tic content mixed with unstructured content in a web-page. • T is a set of tuples (t, Ct), where t is an authoring Dido inherits data exploration capabilities from the underly- technique and Ct a set of possible configurations for ing Exhibit 2 framework. Loomp as a prove-of-concept for the the authoring technique t; One Click Annotation [1] strategy is another example in this • H is a set of helper components. context. Loomp is a WYSIWYG web editor for enriching content with RDFa annotations. It employs a partial mapping between UI elements and data to hide the complexity The WYSIWYM model represents an abstract concept from of creating semantic data. which concrete interfaces can be derived by means of bindings between semantic representation model elements and 3. WYSIWYM CONCEPT configurations of particular UI elements. In this section we introduce the fundamental WYSIWYM concept and formalize key elements of the concept. Formal- izing the WYSIWYM concept has a number of advantages: Definition 2 (Binding). A binding b is a function which First, the formalization can be used as a basis for implemen- maps each element of a semantic representation model e tations (cf. Section 4). Secondly it allows to evaluate and (e ∈ EDi ) to a set of tuples (ui, c), where ui is a user interface technique ui (ui ∈ V ∪ X ∪ T ) and c is a configuration 2 http://simile-widgets.org/exhibit/ c ∈ Cui. resentation models into the three categories tree-based, graph-

high based and hypergraph-based (cf. Figure 2). Each seman-

tic representation model comprises a number of represen- Spatial hypertext tation elements, such as various types of entities and relationships. For visualization, exploration and authoring it is Concept maps of paramount importance to bind the most suitable UI elements to respective representation elements. In the sequel we brieﬂy discuss the three diﬀerent types of representation Hypergraph-based models. Mind-maps Topic Maps Graph-based XTM RDF LTM RDF/XML CTM Tree-based. This is the simplest semantic representation Turtle/N3/N-Triples AsTMa model, where semantics is encoded in a tree-like structure. RDFa Complexity of visual mapping of visual Complexity Tree-based JSON-LD It is suited for representing taxonomic knowledge, such as Microdata

Microformat thesauri, classification schemes, subject heading lists, con- low cept hierarchies or mind-maps. It is used extensively in bi- low Semantic expressiveness high ology and life sciences, for example, in the APG III system (Angiosperm Phylogeny Group III system) of flowering plant classification, as part of the Dimensions of the XBRL (eX- Figure 2: Comparison of existing visual mapping tensible Business Reporting Language) or generically in the techniques in terms of semantic expressiveness and SKOS (Simple Knowledge Organization System). Elements complexity of visual mapping. of tree-based semantic representation usually include: • E1: Item – e.g. Magnoliidae, the item representing all flowering plants. Figure 4 gives an overview on all data model (columns) and • E2: Item type – e.g. biological term for Magnoli- UI elements (rows) and how they can be bound together idae. using a certain configuration (cells). The shades of gray • E3: Item-subitem relationships – e.g. Magnoliidae in a certain cell indicate the suitability of a certain bind- referring to subitem magnolias. ing between a particular UI and data model element. Once • E4: Item property value – e.g. the synonym flowering a selection of data models and UI elements was made and plant for the item Magnoliidae. both are bound to each other encoding a certain configura- • E5: Related items – e.g. the sibling item Eudicots to tion in a binding, we attain a concrete instantiation of our Magnoliidae. WYSIWYM model called WYSIWYM interface. Tree-based data can be serialized as Microdata or Microfor- mats. Definition 3 (WYSIWYM interface). An instantiation of the WYSIWYM model I called WYSIWYM inter- Graph-based. This semantic representation model adds more expressiveness compared to simple tree-based formalisms. face now is a hextuple (DI ,VI ,XI ,TI ,HI , bI ), where: The most prominent representative is the RDF data model, • DI is a selection of semantic representation data mod- which can be seen as a set of triples consisting of subject, els (DI ⊂ D); predicate, object, where each component can be a URI, the • VI is a selection of visualization techniques (VI ⊂ V ); object can be a literal and subject as well as object can be a • XI is a selection of exploration techniques (XI ⊂ X); blank node. The most distinguishing features of RDF from • TI is a selection of authoring techniques (TI ⊂ T ); a simple tree-based model are: the distinction of entities in • HI is a selection of helper components (HI ⊂ H). classes and instances as well as the possibility to express • bI is a binding which binds a particular occurrence of a data model element to a visualization, exploration arbitrary relationships between entities. The graph-based and/or authoring technique; model is suited for representing combinatorial schemes such as concept maps. Graph-based models are used in a very broad range of domains, for example, in the FOAF (Friend Note, that we limit the definition to one binding, which of a Friend) for describing people, their interests and inter- means that only one semantic representation model is sup- connections in a social network, in MusicBrainz to publish ported in a particular WYSIWYM interface at a time. It information about music albums, in the medical domain (e.g. could be also possible to support several semantic represen- DrugBank, Diseasome, ChEMBL, SIDER) to describe the tation models (e.g. RDFa and Microdata) at the same time. relations between diseases, drugs and genes, or generically However, this can be confusing to the user, which is why in the SIOC (Semantically-Interlinked Online Communities) we deliberately excluded this case in our definition. In the vocabulary. Elements of RDF as a typical graph-based data remainder of this sections we discuss the different parts of model are: the WYSIWYM concept in more detail. • E1: Instances – e.g. Warfarin as a drug. • E2: Classes – e.g. anticoagulants drug for Warfarin. • E3: Relationships between entities (instances or classes) 3.1 Semantic Representation Models – e.g. the interaction between Aspirin as an an- Semantic representation models are conceptual data models tiplatelet drug and Warfarin which will increase the to express the meaning of information thereby enabling rep- risk of bleeding. resentation and interchange of knowledge. Based on their • E4: Literal property values – e.g. the halflife for the expressiveness, we can roughly divide popular semantic rep- Amoxicillin. - Highlighting

– E41 : Value – e.g. 61.3 minutes. • V1: Framing and Segmentation (borders, overlays and

– E42 : Language tag – e.g. en. backgrounds). This technique can be applied to text,

– E43 : Datatype – e.g. xsd:float. images and videos, we enclose a semantic entity in a RDF-based data can be serialized in various formats, such coloured border, background or overlay. Different bor- as RDFa, RDF/XML, JSON-LD or Turtle/N3/N-Triples. der styles (colours, width, types), background styles (colours, patterns) or overlay styles (when applied to Hypergraph-based. A hypergraph is a generalization of images and videos) can be used to distinguish differ- a graph in which an edge can connect any number of ver- ent types of semantic entities (cf. Figure 3 no. 1, 2). tices. Since hypergraph-based models allow n-ary relation- The technique is already employed in social network- ships between arbitrary number of nodes, they provide a ing websites such as Google Plus and Facebook to tag higher level of expressiveness compared to tree-based and people within images. graph-based models. The most prominent representative is • V2: Text formatting (color, font, size, etc.). In this the Topic Maps data model developed as an ISO/IEC stan- technique different text styles such as font family, style, dard which consists of topics, associations and occurrences. weight, size, colour, shadows and other text decora- The semantic expressivity of Topic Maps is, in many ways, tion techniques are used to distinguish semantic enti- equivalent to that of RDF, but the major differences are ties within a text (cf. Figure 3 no. 6). The problem that Topic Maps (i) provide a higher level of semantic ab- with this technique is that in an HTML document, straction (providing a template of topics, associations and the applied semantic styles might overlap with exist- occurrences, while RDF only provides a template of two ar- ing styles in the document and thereby add ambiguity guments linked by one relationship) and (hence) (ii) allow to recognizing semantic entities. n-ary relationships (hypergraphs) between any number of • V3: Image color effects. This technique is similar to nodes, while RDF is limited to triplets. The hypergraph- text formatting but applied to images and videos. Dif- based model is suited for representing complex schemes such ferent image color effects such as brightness/contrast, as spatial hypertext. Hypergraph-based models are used for shadows, glows, bevel/emboss are used to highlight se- a variety of applications. Amongst them are musicDNA3 as mantic entities within an image (cf. Figure 3 no. 7). an index of musicians, composers, performers, bands, artists, This technique suffers from the problem that the ap- producers, their music, and the events that link them to- plied effects might overlap with the existing effects in gether, TM4L (Topic Maps for e-Learning), clinical decision the image thereby making it hard to distinguish the support systems and enterprise information integration. El- semantic entities. ements of Topic Maps as a typical hypergraph-based data • V4: Marking (icons appended to text or image). In model are: this technique, which can be applied to text, images and videos, we append an icon as a marker to the part • E1: Topic name – e.g. University of Leipzig. of object which includes the semantic entity (cf. Fig- • E2: Topic type – e.g. organization for University of Leipzig. ure 3 no. 9). The most popular use of this technique is currently within maps to indicate specific points of • E3: Topic associations – e.g. member of a project which has other organization partners. interest. Different types of icons can be used to distinguish different types of semantic or correlated entities. • E4: Topic role in association e.g. coordinator. • V5: Bleeping. A bleep is a single short high-pitched • E5: Topic occurrences – e.g. address. signal in videos. Bleeping can be used to highlight se- – E5 : value – e.g. Augustusplatz 10, 04109 Leipzig. 1 mantic entities within a video. Different type of bleep – E52 : datatype – e.g. text. Topic Maps-based data can be serialized as an XML-based signals can be defined to distinguish different types of syntax called XTM (XML Topic Map), LTM (Linear Topic semantic entities. Map Notation), CTM (Compact Topic Maps Notation) and • V6: Speech (in videos). In this technique a video is aug- AsTMa (Asymptotic Topic Map Notation). mented by some speech indicating the semantic entities and their types within the video. - Associating 3.2 Visualization • V7: Line connectors. Using line connectors is the sim- The primary objectives of visualization are to present, trans- plest way to visualize the relation between semantic form, and convert semantic data into a visual representation, entities in text, images and videos (cf. Figure 3 no. so that, humans can read, query and edit them efficiently. 4). If the value of a property is available in the text, We divide existing techniques for visualization of knowledge line connectors can also reflect the item property val- encoded in text, images and videos into the three categories ues. Problematic is that normal line connectors can Highlighting, Associating and Detail view. Highlighting in- not express the direction of a relation. cludes UI techniques which are used to distinguish or high- • V8: Arrow connectors. Arrow connectors are extended light a part of an object (i.e. text, image or video) from the line connectors with arrows to express the direction of whole object. Associating deals with techniques that visu- a relation in a directed graph. alize the relation between some parts of an object. Detail - Detail view view includes techniques which reveal detailed information • V : Callouts. A callout is a string of text connected by about a part of an object. For each of the above categories, 9 a line, arrow, or similar graphic to a part of text, image the related UI techniques are as follows: or video giving information about that part. It is used in conjunction with a cursor, usually a pointer. The 3http://www.musicdna.info/ Figure 3: Screenshots of user interface techniques for visualization and exploration: 1-framing using borders, 2-framing using backgrounds, 3-video subtitle, 4-line connectors and arrow connectors, 5-bar layouts, 6-text formatting, 7-image color effects, framing and line connectors, 8-expandable callout, 9-marking with icons, 10-tooltip callout, 11-faceting

user hovers the pointer over an item, without click- data will result in faster and easier comprehension of the ing it, and a callout appears (cf. Figure 3 no. 10). targeted content. Techniques for exploration of semantics Callouts come in different styles and templates such encoded in text, images and videos include: as infotips, tooltips, hint and popups. Different sort • X1: Zooming. In a zoomable UI, users can change of semantic information can be embedded in a callout the scale of the viewed area in order to see more de- to indicate the type of semantic entities, property val- tail or less. Zooming in a semantic entity can reveal ues and relationships. Another variant of callouts is further details such as property value or entity type. the status bar which displays semantic information in Zooming out can be employed to reveal the relations a bar appended to the text, image or video container. between semantic entities in a text, image or video. A problem with dynamic callouts is that they do not Supporting rich dynamics by configuring different vi- appear on mobile devices (by hover), since there is no sual representations for semantic objects at different cursor. sizes is a requirement for a zoomable UI. The iMap- • V10: Video subtitles. Subtitles are textual versions of ping approach[6] which is implemented in the semantic the dialog or commentary in videos. They are usually desktop is an example of the zooming technique. displayed at the bottom of the screen and are employed • X2: Faceting. Faceted browsing is a technique for ac- for written translation of a dialog in a foreign lan- cessing information organized according to a faceted guage. Video subtitles can be used to reflect detailed classification system, allowing users to explore a col- semantics embedded in a video scene when watching lection of information by applying multiple filters (cf. the video. A problem with subtitles is efficiently scal- Figure 3 no. 11). Defining facets for each component of ing the text size and relating text to semantic entities the predefined semantic models enable users to browse when several semantic entities exist in a scene. the underlying knowledge space by iteratively narrow- ing the scope of their quest in a predetermined order. One of the main problems with faceted browsers is the 3.3 Exploration increased number of choices presented to the user at To increase the effectiveness of visualizations, users need to each step of the exploration [5]. be capable to dynamically explore the visual representation • X3: Bar layouts. In the bar layout, each semantic en- of the semantic data. The dynamic exploration of semantic tity within the text is indicated by a vertical bar in the authoring. left or right margin (cf. Figure 3 no. 5). The colour • T7: Voice commands. Voice commands permit the of the bar reflects the type of the entity. The bars are user’s hands and eyes to be busy with another task, ordered by length and order in the text. Nested bars which is particularly valuable when users are in motion can be used to show the hierarchies of entities. Seman- or outside. Users tend to prefer speech for functions tic entities in the text are highlighted by a mouse-over like describing objects, sets and subsets of objects [20]. the corresponding bar. This approach is employed in • T8: (Multi-touch) gestures. A gesture (a.k.a. sign Loomp [17]. language) is a form of non-verbal communication in • X4: Expandable callouts. Expandable callouts are in- which visible bodily actions communicate particular teractive and dynamic callouts which enable users to messages. Technically, different methods can be used explore the semantic data associated to a predefined for detecting and identifying gestures. Movement-sensor- semantic entity (cf. Figure 3 no. 8). This technique is based and camera-based approaches are two commonly employed in OntosFeeder [11]. used methods for the recognition of in-air gestures [15]. Multi-touch gestures are another type of gestures which are defined to interact with multi-touch devices such as 3.4 Authoring modern smartphones and tablets. Users can use ges- Semantic authoring aims to add more meaning to digitally tures to determine semantic entities, their types and published documents. If users do not only publish the con- relationship among them. The main problem with ges- tent, but at the same time describe what it is they are pub- tures is their high level of abstraction which makes it lishing, then they have to adopt a structured approach to hard to assert concrete property values. authoring. A semantic authoring UI is a human accessible interface with capabilities for writing and modifying semantic documents. The following techniques can be used for 3.5 Bindings Figure 4 gives an overview on possible bindings between the authoring of semantics encoded in text, images and videos: user interface and semantic representation elements defined • T : Form editing. In form editing, a user employs ex- 1 in previous sections. These possible bindings were obtained isting form elements such as input/check/radio boxes, from a comprehensive survey of existing tools and an anal- drop-down menu, slider, spinner, buttons, date/color ysis of possible connections between a specific UI element picker etc. for content authoring. and a semantic model element. The following binding con- • T : Inline edit. Inline editing is the process of editing 2 figurations are available and refereed to from the cells of items directly in the view by performing simple clicks, Figure 4: rather than selecting items and then navigating to an • Defining a special border or background style (C ), edit form and submitting changes from there. 1 text style (C ), image color effect (C ), beep sound • T : Drawing. Drawing as part of informal user inter- 2 4 3 (C ), bar style (C ), sketch (C ), draggable or drop- faces [14], provides a natural human input to anno- 5 6 7 pable shape (C ), voice command (C ), gesture (C ) tate an object by augmenting the object with human- 8 9 10 or a related icon (C ) for each type. understandable sketches. For instance, users can draw 3 • Progressive shading (C ) by defining continuous shades a frame around semantic entities, draw a line between 11 within a specific color scheme to distinguish items in related entities etc. Special shapes can be drawn to different levels of the hierarchy. indicate different entity types or entity roles in a rela- • Hierarchical bars (C ) by defining special styles for tion. 12 nested bars. • T : Drag and drop. Drag and drop is a pointing device 4 • Grouping by similar border or background style (C ), gesture in which the user selects a virtual object by 13 text style (C ), icons (C ) or image color effects grabbing it and dragging it to a different location or 14 15 (C ). onto another virtual object. In general, it can be used 16 to invoke many kinds of actions, or create various types of associations between two abstract objects. 3.6 Helper Components In order to facilitate, enhance and customize the WYSI- • T5: Context menu. A context menu (also called con- textual, shortcut, or pop-up menu) is a menu that ap- WYM model, we utilize a set of helper components, which pears upon user interaction, such as a right button implement cross-cutting aspects. A helper component acts mouse click. A context menu offers a limited set of as an extension on top of the core functionality of the WYSI- choices that are available in the current state, or con- WYM model. The following components can be used to text. improve the quality of a WYSIWYM UI depending on the requirements defined for a specific application domain: • T6: (Floating) Ribbon editing. A ribbon is a command bar that organizes functions into a series of tabs or • H1: Automation means the provision of facilities for toolbars at the top of the editable content. Ribbon automatic annotation of text, images and videos to re- tabs/toolbars are composed of groups, which are a la- duce the need for human work and thereby facilitating beled set of closely related commands. A floating rib- the efficient annotation of large item collections. For bon is a ribbon that appears when user rolls the mouse example, users can employ existing NLP services (e.g. over a target area. A floating ribbon increases usabil- named entity recognition, relationship extraction) for ity by bringing edit functions as close as possible to automatic text annotation. the user’s point of focus. The Aloha WYSIWYG ed- • H2: Real-time tagging is an extension of automation, itor 4 is an example of floating ribbon based content which allows to create annotations proactively while the user is authoring a text, image or video. This will 4 http://aloha-editor.org significantly increase the annotation speed and users * If value is available in the text/subtitle. Tree-based Graph-based Hypergraph-based (e.g. Taxonomies) (e.g. RDF) (e.g. Topic Maps) No binding Partial binding Full binding

Literal property Topic

Occurre

values nces

type

association

subitem

in in

Item -

Topic

Class

associations

entities Instance

type Item

Topic Topic role role

Item

Related Related Items

Value Value

Datatype Datatype

Topic Topic

Item property property Item value

Relationships Relationships between Language Language tag

Structure Topic UI categories UI techniques encoded in: Framing and segmentation (borders, C1 C11 C13 C1 C1 overlays, backgrounds) Highlighting Text formatting (color, C C C C 2 C font, size etc.) 2 11 14 2

text Marking (appended icons) C3 C15 C3 C3 Line connectors * * * Associating Arrow connectors * * * Callouts Detail view (infotips, tooltips, popups) Framing and segmentation (borders, C1 C11 C13 C1 C1 overlays, backgrounds) Highlighting C4 Image color effects C4 C11 C16 C4

images Marking (appended icons) C3 C15 C3 C3 Line connectors Associating Arrow connectors Callouts Visualization Detail view (infotips, tooltips, popups) Framing and segmentation (borders, C1 C11 C13 C1 C1 overlays, backgrounds)

Image color effects C4 C11 C16 C4 C4 Highlighting C C Marking (appended icons) C 15 3 C 3 3

videos Bleeping C5 C5 C5 Speech Line connectors * * * Associating Arrow connectors * * * Callouts

Detail view (infotips, tooltips, popups) Subtitle Zooming

Faceting text Bar layout C5 C12 C5 C5 Expandable callouts Zooming images Exploration Faceting videos Faceting (excerpts) Form editing Inline edit

Drawing C7 C7 C7 C7 Drag and drop C C C C text, images, 8 8 8 8 videos Context menu

Authoring (Floating) Ribbon editing

Voice commands C9 C9 C9 C9 (Multi-Touch) Gestures C10 C10 C10 C10

Figure 4: Possible bindings between user interface and semantic representation model elements. are not distracted since they do not have to interrupt Usability Factor/Grade Poor Fair Neutral Good Excellent Fit for use 0% 12.50% 31.25% 43.75% 12.50% their current authoring task. Ease of learning 0% 12.50% 50% 31.25% 6.25% • H3: Recommendation means providing users with pre- Task efficiency 0% 0% 56.25% 37.50% 6.25% filled form fields, suggestions (e.g. for URIs, names- Ease of remembering 0% 0% 37.50% 50% 12.50% Subjective satisfaction 0% 18.75% 50% 25% 6.25% paces, properties), default values etc. These facili- Understandability 6.25%18.75% 31.25% 37.50% 6.25% ties simplify the authoring process, as they reduce the number of required user interactions. Moreover, they Table 1: Usability evaluation results for RDFaCE. help preventing incomplete or empty metadata. In order to leverage other user’s annotations as recommendations, approaches like Paragraph Fingerprinting [8] can be implemented. • H4: Personalization and context-awareness describes the ability of the UI to be configured according to users’ contexts, background knowledge and preferences. Instead of being static, a personalized UI dynamically tailors its visualization, exploration and authoring func- tionalities based on the user profile and context. • H5: Collaboration and crowdsourcing enables collabo- rative semantic authoring, where the authoring process can be shared among different authors at different locations. There are a vast amounts of amateur and ex- pert users which are collaborating and contributing on the Social Web. Crowdsourcing harnesses the power of such crowds to significantly enhance and widen the results of semantic content authoring and annotation. Figure 6: Usability evaluation results for Pharmer Generic approaches for exploiting single-user Web ap- (0: Strongly disagree, 1: Disagree, 2: Neutral, 3: plications for shared editing [7] can be employed in this Agree, 4: Strongly agree). context. • H6: Accessibility means providing people with disabil- ities and special needs with appropriate UIs. The un- In order to evaluate RDFaCE usability, we conducted an derlying semantic model in a WYSIWYM UI can allow experiment with 16 participants of the ISSLOD 2011 sum- 6 alternatives or conditional content in different modal- mer school . The user evaluation comprised the following ities to be selected based on the type of the user dis- steps: First, some basic information about semantic content ability and information need. authoring along with a demo showcasing different RDFaCE • H7: Multilinguality means supporting multiple lan- features was presented to the participants as a 3 minutes guages in a WYSIWYM UI when visualizing, exploring video. Then, participants were asked to use RDFaCE to an- or authoring the content. notate three text snippets – a wiki article, a blog post and a news article. For each text snippet, a timeslot of five minutes was available to use different features of RDFaCE for 4. IMPLEMENTATION AND EVALUATION annotating occurrences of persons, locations and organiza- In order to evaluate the WYSIWYM model, we implemented tions with suitable entity references. Subsequently, a survey the two applications RDFaCE and Pharmer, which we present was presented to the participants where they were asked in the sequel. questions about their experience while working with RD- FaCE. Questions were targeting six factors of usability [12] RDFaCE. (RDFa Content Editor) is a WYSIWYM inter- namely Fit for use, Ease of learning, Task efficiency, Ease of face for semantic content authoring. It is implemented on remembering, Subjective satisfaction and Understandability. top of the TinyMCE rich text editor. RDFaCE extends the Results of the survey are shown in Table 1. They indicate on existing WYSIWYG user interfaces to facilitate semantic average good to excellent usability for RDFaCE. A majority authoring within popular CMSs, such as blogs, wikis and of the users deem RDFaCE being fit for use and its function- discussion forums. The RDFaCE implementation (cf. Fig- ality easy to remember. Also, easy of learning and subjective ure 5, left) is open-source and available for download to- satisfaction was well rated by the participants. There was gether with an explanatory video and online demo at http: a slightly lower (but still above average) assessment of task //aksw.org/Projects/RDFaCE. RDFaCE as a WYSIWYM efficiency and understandability, which we attribute to the instantiation can be described using the following hextuple: short time participants had for familiarizing themselves with • D: RDFa, Microdata5. RDFaCE and the quite comprehensive functionality, which • V: Framing using borders (C: special border color de- includes automatic annotations, recommendations and var- fined for each type), Callouts using dynamic tooltips. ious WYSIWYM UI elements. • E: Faceting based on the type of entities. • T: Form editing, Context Menu, Ribbon editing. • H: Automation, Recommendation. Pharmer. Pharmer is a WYSIWYM interface for the au- • b: bindings defined in Figure 4. thoring of semantically enriched electronic prescriptions. It

5Microdata support is implemented in RDFaCE-Lite avail- 6Summer school on Linked Data: http://lod2.eu/ able at http://rdface.aksw.org/lite Article/ISSLOD2011 Figure 5: Screenshots of our two implemented WYSIWYM interfaces. Left: RDFaCE for semantic text authoring (T6 indicates the RDFaCE menu bar, V1 – the framing of named entities in the text, V9 – a callout showing additional type information, T5 – a context menu for revising annotations). Right: Pharmer for authoring of semantic prescriptions (V1 – highlighting of drugs through framing, V9 – additional information about a drug in a callout, T1/T2 combined form and inline editing of electronic prescriptions). enables physicians to embed drug-related metadata into e- we again attribute to the short learning phase and diverse prescriptions thereby reducing the medical errors occurring functionality. in the prescriptions and increasing the awareness of the pa- tients about the prescribed drugs and drug consumption in general. In contrast to database-oriented e-prescriptions, 5. CONCLUSIONS semantic prescriptions are easily exchangeable among other Bridging the gap between unstructured and semantic con- e-health systems without need to changing their related in- tent is a crucial aspect for the ultimate success of semantic frastructure. The Pharmer implementation (cf. Figure 5, technologies. With the WYSIWYM concept we presented right) is open-source and available for download together in this article an approach for integrated visualization, ex- with an explanatory video and online demo7 at http:// ploration and authoring of unstructured and semantic con- code.google.com/p/pharmer/. It is based on HTML5 content. The WYSIWYM model binds elements of a knowl- tenteditable element. Pharmer as a WYSIWYM instantia- edge representation formalism (or data model) to a set of tion is defined using the following hextuple: suitable UI elements for visualization, exploration and au- • D: RDFa. thoring. Based on such a declarative binding mechanism, we • V: Framing using borders and background (C: special aim to increase the flexibility, reusability and development background color defined for each type), Callouts using efficiency of semantics-rich user interfaces dynamic popups. • E: Faceting based on the type of entities. We deem this work as a first step in a larger research agenda • T: Form editing, Inline edit. aiming at improving the usability of semantic user interfaces, • H: Automation, Real-time tagging, Recommendation. while retaining semantic richness and expressivity. In future • b: bindings defined in Figure 4. work we envision to adopt a model-driven approach to enable automatic implementation of WYSIWYM interfaces by user-defined preferences. This will help to reuse, re-purpose In order to evaluate the usability of Pharmer, we performed and choreograph WYSIWYM UI elements to accommodate a user study with 13 subjects. Subjects were 3 physicians, the needs of dynamically evolving information structures 4 pharmacist, 3 pharmaceutical researchers and 3 students. and ubiquitous interfaces. We also aim to bootstrap an We first showed them a 3-minute tutorial video of using dif- ecosystem of WYSIWYM instances and UI elements to sup- ferent features of Pharmer then asked each one to create port structure encoded in different modalities, such as im- a semantic prescription with Pharmer. After finishing the ages and videos. Creating live and context-sensitive WYSI- task, we asked the participants to fill out a questionnaire. WYM interfaces which can be generated on-the-fly based on We used the System Usability Scale (SUS) [13] as a stan- the ranking of available UI elements is another promising re- dardized, simple, ten item Likert scale-based questionnaire search venue. to grade the usability of Pharmer. SUS yields a single number in the range of 0 to 100 which represents a composite measure of the overall usability of the system. The results of our survey (cf. Figure 6) showed a mean usability score Acknowledgments We would like to thank our colleagues from the AKSW re- of 75 for Pharmer WYSIWYM interface which indicates a search group for their helpful comments and inspiring dis- good level of usability. Participants particularly liked the cussions during the development of the WYSIWYM model. integration of functionality and the ease of learning and use. This work was partially supported by a grant from the Eu- The confidence in using the system was slightly lower, which ropean Union’s 7th Framework Programme provided for the 7 http://bitili.com/pharmer project LOD2 (GA no. 257943). 6. REFERENCES [19] B. A. Myers. A brief history of human-computer [1] One Click Annotation. volume 699 of CEUR interaction technology. interactions, 5(2):44–54, 1998. Workshop Proceedings, February 2010. [20] S. Oviatt, P. Cohen, L. Wu, J. Vergo, L. Duncan, [2] A. Ankolekar, M. Krötzsch, T. Tran, and B. Suhm, J. Bers, T. Holzman, T. Winograd, D. Vrandecic. The two cultures: mashing up web 2.0 J. Landay, J. Larson, and D. Ferro. Designing the user and the semantic web. In WWW 2007, pages 825–834. interface for multimodal speech and pen-based gesture [3] G. Burel, A. E. Cano1, and V. Lanfranchi. Ozone applications: state-of-the-art systems and future browser: Augmenting the web with semantic overlays. research directions. Hum.-Comput. Interact., volume 449 of CEUR WS Proceedings, June 2009. 15(4):263–322, Dec. 2000. [4] A.-S. Dadzie and M. Rowe. Approaches to visualising [21] E. Pietriga, C. Bizer, D. R. Karger, and R. Lee. linked data: A survey. Semantic Web, 2(2):89–124, Fresnel: A browser-independent presentation 2011. vocabulary for rdf. In ISWC, LNCS, pages 158–171. [5] L. Deligiannidis, K. J. Kochut, and A. P. Sheth. RDF Springer, 2006. data exploration and visualization. In CIMS 2007, [22] R. Power, R. Power, D. Scott, D. Scott, R. Evans, and pages 39–46. ACM, 2007. R. Evans. What you see is what you meant: direct [6] H. Haller and A. Abecker. imapping: a zooming user knowledge editing with natural language feedback, interface approach for personal and semantic 1998. knowledge management. SIGWEB Newsl., pages [23] M. Sah, W. Hall, N. M. Gibbins, and D. C. D. Roure. 4:1–4:10, Sept. 2010. Semport - a personalized semantic portal. In 18th [7] M. Heinrich, F. Lehmann, T. Springer, and ACM Conf. on Hypertext and Hypermedia, pages M. Gaedke. Exploiting single-user web applications for 31–32, 2007. shared editing: a generic transformation approach. In [24] C. Sauer. What you see is wiki – questioning WWW 2012, pages 1057–1066. ACM, 2012. WYSIWYG in the Internet age. In Proceedings of [8] L. Hong and E. H. Chi. Annotate once, appear Wikimania 2006, 2006. anywhere: collective foraging for snippets of interest [25] B. Shneiderman. Creating creativity: user interfaces using paragraph fingerprinting. CHI ’09, pages for supporting innovation. ACM Trans. 1791–1794. ACM. Comput.-Hum. Interact., 7(1):114–138, Mar. 2000. [9] D. R. Karger, S. Ostler, and R. Lee. The web page as [26] J. Spiesser and L. Kitchen. Optimization of html a wysiwyg end-user customizable database-backed automatically generated by wysiwyg programs. In information management application. In UIST 2009, WWW 2004, pages 355–364. pages 257–260. ACM, 2009. [27] D. Tunkelang. Faceted Search (Synthesis Lectures on [10] A. Khalili and S. Auer. User interfaces for semantic Information Concepts, Retrieval, and Services). authoring of textual content: A systematic literature Morgan and Claypool Publishers, June 2009. review. 2012. [11] A. Klebeck, S. Hellmann, C. Ehrlich, and S. Auer. Ontosfeeder - a versatile semantic context provider for web content authoring. In ISWC 2011, volume 6644 of LNCS, pages 456–460. [12] S. Lauesen. User Interface Design: A Software Engineering Perspective. Addison Wesley, Feb. 2005. [13] J. Lewis and J. Sauro. The Factor Structure of the System Usability Scale. In Human Centered Design, volume 5619 of LNCS, pages 94–103. 2009. [14] J. Lin, M. Thomsen, and J. A. Landay. A visual language for sketching large and complex interactive designs. CHI ’02, pages 307–314. ACM. [15] A. Loecken, T. Hesselmann, M. Pielot, N. Henze, and S. Boll. User-centred process for the definition of free-hand gestures applied to controlling music playback. Multimedia Syst., 18(1):15–31, 2012. [16] V. Lopez, V. Uren, M. Sabou, and E. Motta. Is question answering fit for the semantic web? a survey. Semantic Web ? Interoperability, Usability, Applicability, 2(2):125–155, September 2011. [17] R. H. M. Luczak-Roesch. Linked data authoring for non-experts. In WWW WS on Linked Data on the Web (LDOW2009), 2009. [18] W. Muller, I. Rojas, A. Eberhart, P. Haase, and M. Schmidt. A-r-e: The author-review-execute environment. Procedia Computer Science, 4:627 – 636, 2011. ICCS 2011.