Wright State University CORE Scholar
The Ohio Center of Excellence in Knowledge- Kno.e.sis Publications Enabled Computing (Kno.e.sis)
2010
Computing for the Human Experience: Semantics-Empowered Sensors, Services, and Social Computing on the Ubiquitous Web
Amit P. Sheth Wright State University - Main Campus, [email protected]
Follow this and additional works at: https://corescholar.libraries.wright.edu/knoesis
Part of the Bioinformatics Commons, Communication Technology and New Media Commons, Databases and Information Systems Commons, OS and Networks Commons, and the Science and Technology Studies Commons
Repository Citation Sheth, A. P. (2010). Computing for the Human Experience: Semantics-Empowered Sensors, Services, and Social Computing on the Ubiquitous Web. IEEE Internet Computing, 14 (1), 88-91. https://corescholar.libraries.wright.edu/knoesis/543
This Article is brought to you for free and open access by the The Ohio Center of Excellence in Knowledge-Enabled Computing (Kno.e.sis) at CORE Scholar. It has been accepted for inclusion in Kno.e.sis Publications by an authorized administrator of CORE Scholar. For more information, please contact [email protected]. Amit Sheth, "Computing for Human Experience: Semantics empowered Sensors, Services, and Social Computing on Ubiquitous Web," in IEEE Internet Computing - Vision Issue (V. Cerf and M. Singh, Eds.), vol. 14, no. 1, pp. 88-91, Jan./Feb. 2010. Semantics & Services Computing for Human Experience Semantics-Empowered Sensors, Services, and Social Computing on the Ubiquitous Web
Amit Sheth • Wright State University
n his influential paper “The Computer for focusing on humans interacting with a technol- the 21st Century,” Mark Weiser talked about ogy or system, CHE will feature technology-rich I making machines fit the human environ- human surroundings that often initiate interac- ment instead of forcing humans to enter the tions. Interaction will be more sophisticated and machine’s environment.1 He noted, “The most seamless compared to today’s precursors such as profound technologies are those that disap- automotive accident-avoidance systems. pear. They weave themselves into the fabric of Many components of and ideas associated everyday life until they are indistinguishable with the CHE vision have been around for a from it.” Weiser’s vision, outlined two decades while. Here, I discuss some of the most impor- ago, led to ubiquitous computing. Now, we must tant tipping points that I believe will make CHE again rethink the relationship and interactions a reality within a decade. between humans and machines — this time, including a variety of technologies, includ- Bridging the Physical/Digital Divide ing computing technologies; communication, We’ve already seen significant progress in tech- social-interaction, and Web technologies; and nology that enhances human-computer inter- embedded, fixed, or mobile sensors and devices. actions; the iPhone is a good example. Now We’re on the verge of an era in which the we’re seeing increasingly intelligent interfaces, human experience can be enriched in ways we as exemplified by Tom Gruber’s Intelligence at couldn’t have imagined two decades ago. Rather the Interface technology (http://tomgruber.org/ than depending on a single technology, we’ve pro- news/sdforum-dec13.htm), which has demon- gressed with several whose semantics-empowered strated contextual use of knowledge to develop convergence and integration will enable us to intelligent human–mobile-device interfaces. We’re capture, understand, and reapply human knowl- also seeing progress in how machines (devices edge and intellect. Such capabilities will conse- and sensors), surroundings, and humans inter- quently elevate our technological ability to deal act, enabled by advances in sensing the body, with the abstractions, concepts, and actions that the mind, and place. Such research supports the characterize human experiences. This will herald ability to understand human actions, including computing for human experience (CHE). human gestures and languages in increasingly The CHE vision is built on a suite of tech- varied forms. The broadening ability to give any nologies that serves, assists, and cooperates physical object an identity in the cyber world with humans to nondestructively and unobtru- (that is, to associate the object with its repre- sively complement and enrich normal activi- sentation), as contemplated with the Internet ties, with minimal explicit concern or effort of Things, will let machines leverage extensive on the humans’ part. CHE will anticipate when knowledge about the object to complement what to gather and apply relevant knowledge and humans process. intelligence. It will enable human experiences Human-machine interactions are taking that are intertwined with the physical, concep- place at a new level, as demonstrated by Psyler- tual, and experiential worlds (emotions, senti- on’s Mind Lamp (www.psyleron.com/lamp.aspx), ments, and so on), rather than immerse humans which shows connections between the mind in cyber worlds for a specific task. Instead of and the physical world. Soon, computers will be
88 Published by the IEEE Computer Society 1089-7801/10/$26.00 © 2010 IEEE IEEE INTERNET COMPUTING © 2010 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE. Computing for Human Experience
able to translate gestures to concrete personality account for much of the will all come together to enable a actionable cues and understand per- background knowledge in a single system that makes conclusions and ceptions behind human observations, mind. Machines also must leverage decisions with human-like intuition, as shown by MIT’s Sixth Sense proj- background knowledge for effective but much more quickly than humans. ect (www.pranavmistry.com/projects/ perception.3 So, effective perception sixthsense). requires a framework for represent- Semantics at an In addition, interactions initi- ing background knowledge. Extraordinary Scale ated in the cyber world are increas- Semantic computing, aided by ing and becoming richer. Examples From Perception Semantic Web technology, is an range from a location-aware system to Semantics ideal candidate framework for mean- telling a smart phone user about a John Locke, Charles Peirce, Ber- ingful representation and sharing of sale item’s availability nearby to trand Russell, and many others have hypotheses and background knowl- advanced processing of sensor data extensively and wonderfully written edge. Together with semantic com- and crowd intelligence to recom- about semiotics — how we construct puting, the large-scale adoption mend a road rerouting or to act on and understand meaning through of Web 2.0 or social-Web technol- behalf of a human. This bridging of symbols. A key enhancement we’re ogy has led to the availability of the physical/digital divide is a key already seeing is the humanization multimodal user-generated content, part of CHE. of data and observation, includ- whether text, audio, video, or sim- ing social computing extending ply attention metadata, from a vari- Elevating Abstractions semantic computing and vice versa. ety of online networks. The most That Machines Understand Metadata is no longer confined to promising aspect of this data is Perception is a key aspect of human structural, syntactic, and seman- that it truly represents a population intelligence and experience. Elevating tic metadata but includes units of and isn’t a biased response or arbi- machine perception to a level closer observations that convey human trary sample study. This means that to that of human perception will be experience, including perceptions, machines now have at their disposal a key enabler of CHE. In 1968, Rich- sentiments, opinions, and intentions. the variety and vastness of data and ard Gregory described perception as Soon, we’ll be able to convert the local and global contexts that a hypothesis over observation.2 Such massive amounts of raw data and we use in our day-to-day processing hypothesis building comes naturally observations into symbolic representa- of information to gather insights or to people as an (almost) entirely sub- tions. We’ll make these representations make decisions. conscious activity. Humans often more meaningful through a variety of We also see a move from document- interpret the raw sensory observa- relationships and associations we can and keyword-centric information pro- tion before recognizing a conscious establish with other things we know. cessing that relies on search-and-sift thought. On the other hand, hypoth- We’ll then be able to contextually to representing information at higher esis building is often cumbersome leverage all this to improve human abstraction levels. This involves mov- for machines. Nevertheless, to inte- activities and experience. ing from entity- or object-centric grate human and machine percep- CHE will bring together many processing to relationship- and event- tion, the convergence must occur at current technological advances in centric processing. This, in turn, this abstraction level, often termed capabilities that are easy and natural involves improving the ability to situation awareness. Therefore, for humans but harder for machines, extract, represent, and reason about regardless of the source, this inte- fundamentally combining human a vast variety of relationships, as gration requires a shared framework sensing with machine sensing and well as providing integral support for for communicating and comparing processing. In CHE, information’s temporal, spatial, and situation awareness. thematic elements. Perceptual hypotheses represent • pattern recognition, With parallel advances in knowl- the semantics, or meaning, of obser- • image analysis, edge engineering, large-scale data vation. Beginning with raw obser- • casual text processing, analytics, and language understand- vation, we find such meaning by • sentiment and intent detection, ing, we’re able to build systems that leveraging background knowledge • using domain models to gather can process, represent, and reason of the interaction between observa- factual information, and over data points much as humans tion and possible causes to deter- • polling social media to gather do. In addition, we can provide mine the most likely hypothesis. community opinions and build extremely rich markups of all the Previous experience, schooling, and intelligence observations available to a machine,
JANUARY/FEBRUARY 2010 89 Semantics & Services
Search Integration Analysis example, Linked Data; http:// Discovery Question answering linkeddata.org), • data and information contributed Relationship Web Situational awareness Patterns/inference/reasoning by a community of volunteers (for example, Wikipedia) and the Metadata and record of social discourse of mil- semantic lions of users on a vast number of annotations topics (for example, Facebook and Metadata extraction Twitter), and • the collection of observations from sensors in, on, or around Conceptual humans; and around the earth. models and Structured and Variety of text — Multimedia Sensor data background semistructured scienti c literature, content Semantic computing over such a knowledge data news, social media content, and so on bewildering variety of data is made possible largely by an agreement on Figure 1. A contemporary architecture for semantic computing. It analyzes what the data means. This agree- any form of Web, social, or sensor data by extracting metadata, resulting ment can be represented in a manner in comprehensive semantic annotation. This process is aided by conceptual that’s formal or informal; explicit or models and knowledge and by a variety of information-retrieval, statistical, and implicit; or static (through a deliber- AI (machine learning and natural-language processing) techniques, at the Web ate, expert-driven process), periodic, scale. Semantic analysis supported by mining, inferencing, and reasoning over or dynamic (for example, mining annotations supports applications. Wikipedia to extract a targeted tax- onomy). The key forms in which such agreements are modeled include for- letting machines connect the dots Semantic Computing mal ontologies, folksonomies, taxon- (contexts) surrounding the observa- as a Starting Point omies, vocabularies, and dictionaries. tions (data) and draw conclusions At the center of the approach to Researchers have started making that nearly mimic human perception achieving CHE is semantic comput- rapid strides in creating models and and cognition. All these together ing. Figure 1 shows a contemporary background knowledge from human are reducing the disparity between architecture for semantic computing collaborations (as exemplified by humans’ perceptions and the con- (simplified for brevity). It has four hundreds of expert-created ontolo- clusions that machines draw from key components: data or resources, gies). They’ve done this by quantified or qualified observations. models and knowledge, semantic Semantics-empowered social com annotation, and semantic analysis • selectively extracting or min- puting, semantics-empowered ser or reasoning. ing the Web for facts, as demon- vices computing (smart mashups), Whereas semantic computing strated by Voquette/Semangix,4 and semantics-enhanced sen- started with primarily enterprise or scientific literature and sor computing (exemplified by the and then Web data, including busi- • harvesting community-created semantic sensor Web) are key build- ness and scientific data and litera- content, as demonstrated by ing blocks of CHE. ture, it has expanded to include any Yago (www.mpi-inf.mpg.de/yago The sidebar “Influential and Inter- type of data and massive amounts -naga/yago) and Taxonom (http:// esting Works That Lead to Computing of Web-accessible resources, includ- taxonom.com). for Human Experience” (available at ing services, sensor, and social data. www.computer.org/cms/Computer. Here are some impressive examples: Along with domain-specific or org/dl/mags/ic/2010/01/extras/ thematic conceptual models, tempo- mic2010010088s.pdf) discusses such • the capture of comprehensive ral and spatial models (ontologies) important research as the memex personal information (for exam- have taken their rightful place for and trailblazing, ambient intelli- ple, MyLifebits; http://research. capturing meaning, especially as we gence, the Semantic Web, experi- microsoft.com/en-us/projects/ seek to go from keywords and docu- ential computing, the Relationship mylifebits), ments to objects, and then to rela- Web, humanist computing, and the • the collection of massive amounts tionships and events. PeopleWeb. of interlinked curated data (for Such models and associated back
90 www.computer.org/internet/ IEEE INTERNET COMPUTING Computing for Human Experience
Semantic annotation of SWE
JANUARY/FEBRUARY 2010 91 http://www.computer.org/cms/Computer.org/dl/mags/ic/2010/01/extras/mic2010010088s.pdf web extra
Semantics & Services In uential and Interesting Works That Lead to Computing for Human Experience
Amit Sheth, Wright State University
V. Bush, “As We May Think,” The Atlantic, July 1945; www. 1999. Chapter 12 of the book introduced the Semantic Web, theatlantic.com/doc/194507/bush. A true visionary, Vanne- which uses metadata to extensively associate meaning with var Bush presented interesting ideas on the memex (a theoreti- data. A 2000 keynote I gave built on the data/information/ cal computer system) and trailblazing. This is one of the early knowledge-centric work performed in the 1990s (see http:// and important writings emphasizing the role of relationships, knoesis.wright.edu/library/resource.php?id=735). In that key- which has inspired much of my research. Cartic Ramakrishnan note, I discussed uses of ontologies and semantic metadata ex- and I further explored an aspect of this vision in “Relationship traction and annotation that led to commercial applications Web: Blazing Semantic Trails between Web Resources,” IEEE in semantic categorization, cataloging, search, browsing, per- Internet Computing, vol. 11, no. 4, 2007, pp. 84–88. sonalization, and targeting. Berners-Lee (along with James Hendler and Ora Lassila) followed up Weaving the Web with E. Zelkha and B. Epstein, “From Devices to ‘Ambient Intel- a more detailed intelligent-agent, AI-centric vision in the ligence: The Transformation of Consumer Electronics,’” talk highly cited May 2001 Scienti c American article “The Se- given at the 1998 Digital Living Room Conference (Pow- mantic Web.” The Semantic Web initiative carried out in the erPoint presentation available at www.epstein.org/brian/ W3C’s context has led to the development of standards (for ambient_intelligence/DLR%20Final%20Internal.ppt). This example, RDF and OWL) resulting in Web-scale data integra- vision of digital environments that are sensitive and responsive tion and sharing, and knowledge aggregation (for example, to the presence of people goes beyond ubiquitous computing using ontologies) and application. CHE envisages extending and advanced human-computer interaction. In this vision, de- these unprecedented semantic data and computing capabilities vices work together to help people easily and naturally per- to sensory, social, and experiential aspects. form everyday activities. To do this, the devices employ infor- mation and intelligence hidden in the network connecting the Scientists have recognized semantics’ importance in dealing devices, using context-aware, anticipatory, and adaptive tech- with data heterogeneity since the 1980s, when conceptual or nologies. Computing for Human Experience (CHE) also seeks semantic models emerged. The 1990s through the early 2000s to enhance this vision beyond its device roots. CHE leverages saw the use of conceptual models or ontologies for semantic additional progress in perceptual, semantic, and social capa- metadata extraction and annotations, for faceted or semantic bilities, emphasizing experiential attributes. It spans the scope search, and subsequently for semantic analytics. Some early from an individual and his or her surroundings to the collec- examples of semantics in computing appear in J. He¦ in and tive social consciousness. J. Hendler, “A Portrait of the Semantic Web in Action,” IEEE Intelligent Systems, vol. 16, no. 2, 2001, pp. 54–59, and A. T. Berners-Lee and M. Fischetti, Weaving the Web, Harper, Sheth and C. Ramakrishnan, “Semantic (Web) Technology in
Unless otherwise stated, bylined articles, as well as product and service descriptions, re ect the author’s or rm’s opinion. Inclusion does not necessarily constitute endorsement by the IEEE or the IEEE Computer Society.
JANUARY/FEBRUARY 2010 © 2010 IEEE IEEE INTERNET COMPUTING W E B E X T R A i web extra
Action: Ontology Driven Information Systems for Search, In- increasing ability to observe or sense a vast variety of physical tegration and Analysis,” IEEE Data Engineering Bull., vol. as well as social phenomena and events through the emerging 26, no. 4, 2003, pp. 40–48. An example of the extension of sensor Web and social Web. CHE will understand, integrate, semantic computing to today’s data explosion appears in C. analyze, and utilize all these to enhance human experience. Bizer, T. Heath, and T. Berners-Lee, “Linked Data—the Story So Far,” Int’l J. Semantic Web & Information Systems, vol. 5, J. Rossiter, “Humanist Computing: Modelling with Words, no. 2, 2009, pp. 1–22. Concepts, and Behaviours,” Modelling with Words, LNCS 2873, Springer, 2003, pp. 124–152. This paper is relevant to R. Kurzweil, The Age of Spiritual Machines: When Comput- CHE’s focus on raising the level of abstraction. It proposes ers Exceed Human Intelligence, Penguin, 2000. That the pace modeling with words, concepts, and behaviors to deª ne a hi- of technological innovation is accelerating is believable. How- erarchy of methods that extends from low-level data-driven ever, the prospect of computers exceeding human intelligence, modeling with words to the high-level fusion of knowledge in as Ray Kurzweil postulates, is (whether likely or not) certainly the context of human behaviors. less exciting to us humans, in my opinion. R. Ramakrishnan and A. Tomkins, “Toward a PeopleWeb,” R. Jain, “Experiential Computing,” Comm. ACM, vol. 46, no. Computer, vol. 40, no. 8, 2007, pp. 63–72. This article and 7, 2003, pp. 48–55. This article focuses on incorporating in- numerous other works (including Wikipedia’s Collective Intel- sight and experiences into computing, supported by users ap- ligence entry and my article “Citizen Sensing, Social Signals, plying their senses to all forms of data related to events. Also and Enriching Human Experience,” IEEE Internet Comput- relevant are Ramesh Jain’s views on EventWeb; see “Event- ing, vol. 13, no. 4, 2009, pp. 87–92) discuss how connecting Web: Developing a Human-Centered Computing System,” billions of people, generating content based on massive num- Computer, vol. 41, no. 2, 2008, pp. 42–50. CHE is partly a bers of user contributions, citizen or participatory sensing, continuation of the experiential-computing vision. It builds on and using the Web to capture social consciousness are chang- advances in information and communication technology, sen- ing how we collect, share, and analyze human knowledge and sors, and embedded computing, incorporating social and se- experience. To realize CHE, we’ll need to combine physical, mantic computing and emphasizing higher-level abstractions. personal, social, collective, and environmental scopes. Jain and I coedit the Springer book series Semantic and Be- yond: Computing for Human Experience, in which we ª rst This Web extra accompanies the article, “Computing for Hu- conceived the term “computing for human experience.” man Experience: Semantics-Empowered Sensors, Services, and Social Computing on the Ubiquitous Web, by Amit Sheth, IEEE J. Gemmell, G. Bell, and R. Lueder, “MyLifeBits: A Personal Internet Computing, vol. 14, no. 1, 2010, pp. 88–91; http:// Database for Everything,” Comm. ACM, vol. 49, no. 1, 2006, doi.ieeecomputersociety.org/10.1109/MIC.2010.4. pp. 88–95. MyLifeBits presages an ability to record virtually everything in a person’s life. Such an ability to capture per- Selected CS articles and columns are also available for sonal and related social information is complemented by the free at http://ComputingNow.computer.org.
u.
news to yo
Let us bring technology
http://computingnow
Subscribe to our daily newsfeed .computer
.org
ii IEEE INTERNET COMPUTING W E B E X T R A WWW.COMPUTER.ORG/INTERNET/