DOCSLIB.ORG

Automatic Online Evaluation of Intelligent Assistants

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Automatic Online Evaluation of Intelligent Assistants Automatic Online Evaluation of Intelligent Assistants Jiepu Jiang1*, Ahmed Hassan Awadallah2, Rosie Jones2, Umut Ozertem2, Imed Zitouni2, Ranjitha Gurunath Kulkarni2 and Omar Zia Khan2 1 Center for Intelligent Information Retrieval, University of Massachusetts Amherst 2 Microsoft Redmond, WA USA [email protected], {hassanam, rosie.jones, umuto, izitouni, raguruna, omarzia.khan}@microsoft.com ABSTRACT These applications are usually evaluated by comparing system-gen- Voice-activated intelligent assistants, such as Siri, Google Now, erated answers with “correct” answers labeled by human annota- and Cortana, are prevalent on mobile devices. However, it is chal- tors. For example, in web search, we annotate relevant webpages lenging to evaluate them due to the varied and evolving number of and evaluate using metrics such as mean average precision (MAP) tasks supported, e.g., voice command, web search, and chat. Since and normalized discounted cumulative gain (nDCG) [14]. each task may have its own procedure and a unique form of correct However, intelligent assistants differ from these applications in answers, it is expensive to evaluate each task individually. This pa- that they can involve a wide variety of tasks, ranging from making per is the first attempt to solve this challenge. We develop con- phone calls and managing calendars, to finding places, finding an- sistent and automatic approaches that can evaluate different tasks swers to general questions, and web search. These tasks have dif- in voice-activated intelligent assistants. We use implicit feedback ferent forms of “correct” answers. It is expensive to evaluate each from users to predict whether users are satisfied with the intelligent task separately using different human judgments and metrics. It is assistant as well as its components, i.e., speech recognition and in- also difficult to use one single setup to evaluate all tasks. In addi- tent classification. Using this approach, we can potentially evaluate tion, the tasks performed can be personal in nature and the perfor- and compare different tasks within and across intelligent assistants mance of the system depends heavily on users. These factors make according to the predicted user satisfaction rates. Our approach is it challenging to conduct manual ground-truth-based evaluation. characterized by an automatic scheme of categorizing user-system To solve these challenges, we adopt approaches similar to recent interaction into task-independent dialog actions, e.g., the user is studies of user satisfaction prediction in web search [1, 3, 6, 7, 17, commanding, selecting, or confirming an action. We use the action 32]. These studies developed alternative evaluation approaches by sequence in a session to predict user satisfaction and the quality of finding and using correlation between explicit ratings of user expe- speech recognition and intent classification. We also incorporate rience and implicit behavioral signals such as click and dwell time other features to further improve our approach, including features [4, 7]. However, we cannot simply apply user behavior signals in derived from previous work on web search satisfaction prediction, web search to evaluate intelligent assistants due to the wider range and those utilizing acoustic characteristics of voice requests. We and different nature of tasks. These tasks may involve a variety of evaluate our approach using data collected from a user study. Re- user intents, diverse topics, and distinct user interaction modes. For sults show our approach can accurately identify satisfactory and un- example, the process of making a phone call, or navigating to a satisfactory sessions. place, involves a dialog style conversation between user and sys- tem, with user requests and system responses very different from Categories and Subject Descriptors those in web search. In addition, intelligent assistants heavily use H.5.2 [Information Interfaces and Presentation]: User Interfaces voice interactions, and it is important to consider the voice signal – evaluation/methodology, interaction styles, voice I/O. to assess user experience. We introduce a model for evaluating user experience in voice- Keywords activated intelligent assistants. We consider satisfaction as the ma- Voice-activated intelligent assistant; evaluation; user experience; jor indicator of user experience because our study shows that it is mobile search; spoken dialog system. consistently correlated with changes in user interests towards the 1. INTRODUCTION system. Our model predicts whether the user has satisfactory or un- Intelligent assistants are becoming a prevalent feature on mobile satisfactory experience with an intelligent assistant based on user devices. They provide voice control and feedback to mobile device interaction patterns. Once the model is trained, it can evaluate real functions (e.g., making phone calls, calendar management, finding traffic of intelligent assistants without human judgments of correct places). Users can also search the web or even chat with intelligent answers for tasks. This makes it a useful and cheap evaluation ap- assistants. While these novel applications are useful and attractive proach for intelligent assistants’ developers, who have abundant for users, it is challenging to evaluate and compare them due to the user traffic and user logs. Our model includes a sub-model for au- large variability of tasks. tomatically classifying user-system interaction into dialog actions, Evaluation is a central component of many related applications, a Markov model over action transitions, as well as features related e.g., search engines, Q&A systems, and recommendation systems. to requests, responses, clicks, and those using acoustic signals. Our contributions can be summarized as follows: * Work done during an internship at Microsoft Research • An accurate model for predicting user satisfaction with an intel- ligent assistant and its components, i.e., speech recognition and Copyright is held by the International World Wide Web Conference Com- intent classification. mittee (IW3C2). IW3C2 reserves the right to provide a hyperlink to the au- • A scheme of categorizing user-system interaction into task-inde- thor’s site if the Material is used in electronic media. pendent dialog actions, and a model to automatically map differ- WWW 2015, May 18–22, 2015, Florence, Italy. ACM 978-1-4503-3469-3/15/05. ent actions to this scheme. http://dx.doi.org/10.1145/2736277.2741669 • Analysis of user behavior and patterns indicating user experience in intelligent assistants. 2. RELATED WORK itself as a solid foundation for managing dialogues, and a compre- There are a number of areas of related work relevant to the research hensive review can be found in [33]. described in this paper. These include (1) methods and metrics for Since they also support other forms of interactions, intelligent the evaluation of search systems, (2) inferring satisfaction from ob- assistants differ from traditional spoken dialog systems. In addition served search behavior and (3) dialog act modeling and classifica- to voice system response, intelligent assistants provide answers or tion in conversational speech. We cover these in turn in this section. options in the display, and users can type in the requests and select User behavior modeling has been used extensively for evaluating a displayed result or option. In this sense, intelligent assistants are search systems [1, 4, 6, 7]. Traditionally search system have been related to multi-modal conversational systems [11, 20, 30]. evaluated using retrieval metrics such as MAP and nDCG [14], Note that many different taxonomies of dialog acts have been where a collection of documents, queries and human labeled rele- proposed [28]. We do not intend here to propose a new one, but vance judgments are used to evaluate search system performance. rather to model user and system interaction with the goal of pre- These metrics are expensive to collect and potentially noisy, given dicting user satisfaction. Our model of system interaction and user that third-party judges have limited knowledge of the individual is designed independently from any dialog model the system uses. user’s intent. Additionally, these metrics are query-based. Previous Hence, it is independent of any specific implementation. In contrast research has shown that search tasks often contain multiple queries with work on learning dialog model transitions [27] we do not at- related to the same information need [7]. Unfortunately the connec- tempt to model the most likely dialog sequence, but to use the dia- tions between these queries are ignored by these metrics. Session- log sequence to model user satisfaction. Our work differs from pre- based DCG (sDCG) [15] does consider the session-context, but still vious work in offline evaluation of dialog systems [31], as we do requires manual relevance judgments. not require manual transcription of speech, and thus once trained, Another line of research has focused on using implicit feedback our models can be run online, at-scale, evaluating voice assistants from user behavior to evaluate search engine. These methods have in an automated, unsupervised fashion. lower cost, are more scalable, and sourced from the actual users. 3. INTELLIGENT ASSISTANTS Early research on implicit feedback [4] used an instrumented Intelligent assistants are emerging and evolving applications lack- browser to determine if there was an association between explicit ing a precise definition.
Load more

Recommended publications
  • Adobe Introduction to Scripting
    ADOBE® INTRODUCTION TO SCRIPTING © Copyright 2007 Adobe Systems Incorporated. All rights reserved. Adobe® Introduction to Scripting NOTICE: All information contained herein is the property of Adobe Systems Incorporated. No part of this publication (whether in hardcopy or electronic form) may be reproduced or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written consent of Adobe Systems Incorporated. The software described in this document is furnished under license and may only be used or copied in accordance with the terms of such license. This publication and the information herein is furnished AS IS, is subject to change without notice, and should not be construed as a commitment by Adobe Systems Incorporated. Adobe Systems Incorporated assumes no responsibility or liability for any errors or inaccuracies, makes no warranty of any kind (express, implied, or statutory) with respect to this publication, and expressly disclaims any and all warranties of merchantability, fitness for particular purposes, and non-infringement of third-party rights. Any references to company names in sample templates are for demonstration purposes only and are not intended to refer to any actual organization. Adobe®, the Adobe logo, Illustrator®, InDesign®, and Photoshop® are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States and/or other countries. Apple®, Mac OS®, and Macintosh® are trademarks of Apple Computer, Inc., registered in the United States and other countries. Microsoft®, and Windows® are either registered trademarks or trademarks of Microsoft Corporation in the United States and other countries. JavaScriptTM and all Java-related marks are trademarks or registered trademarks of Sun Microsystems, Inc.
    [Show full text]
  • Vbscript Programmer’S Reference Third Edition
    VBScript Programmer’s Reference Third Edition Adrian Kingsley-Hughes Kathie Kingsley-Hughes Daniel Read Wiley Publishing, Inc. ffirs.indd iii 8/28/07 9:41:21 AM ffirs.indd vi 8/28/07 9:41:22 AM VBScript Programmer’s Reference Third Edition Introduction . xxv Chapter 1: A Quick Introduction to Programming . 1 Chapter 2: What VBScript Is — and Isn’t! . 31 Chapter 3: Data Types . 45 Chapter 4: Variables and Procedures . 83 Chapter 5: Control of Flow . 109 Chapter 6: Error Handling and Debugging . 129 Chapter 7: The Scripting Runtime Objects . 183 Chapter 8: Classes in VBScript (Writing Your Own COM Objects) . 209 Chapter 9: Regular Expressions . 233 Chapter 10: Client-Side Web Scripting . 261 Chapter 11: Windows Sidebars and Gadgets . 287 Chapter 12: Task Scheduler Scripting . 309 Chapter 13: PowerShell . 345 Chapter 14: Super-Charged Client-Side Scripting . 375 Chapter 15: Windows Script Host . 405 Chapter 16: Windows Script Components . 465 Chapter 17: Script Encoding . 489 Chapter 18: Remote Scripting . 509 Chapter 19: HTML Applications . 517 Chapter 20: Server-Side Web Scripting . 535 Chapter 21: Adding VBScript to Your VB and .NET Applications . 569 (Continued) ffirs.indd i 8/28/07 9:41:21 AM Appendix A: VBScript Functions and Keywords . 603 Appendix B: Variable Naming Convention . 675 Appendix C: Coding Conventions . 677 Appendix D: Visual Basic Constants Supported in VBScript . 681 Appendix E: VBScript Error Codes and the Err Object . 687 Appendix F: The Scripting Runtime Library Object Reference . 703 Appendix G: The Windows Script Host Object Model . 715 Appendix H: Regular Expressions . 723 Appendix I: The Variant Subtypes .
    [Show full text]
  • Level One Benchmark Windows NT 4.0 Operating Systems V1.0.5
    Level One Benchmark Windows NT 4.0 Operating Systems V1.0.5 Copyright 2003, The Center for Internet Security www.cisecurity.org Page 2 of 32 Terms of Use Agreement Background. CIS provides benchmarks, scoring tools, software, data, information, suggestions, ideas, and other services and materials from the CIS website or elsewhere (“Products”) as a public service to Internet users worldwide. Recommendations contained in the Products (“Recommendations”) result from a consensus-building process that involves many security experts and are generally generic in nature. The Recommendations are intended to provide helpful information to organizations attempting to evaluate or improve the security of their networks, systems and devices. Proper use of the Recommendations requires careful analysis and adaptation to specific user requirements. The Recommendations are not in any way intended to be a “quick fix” for anyone’s information security needs. No representations, warranties and covenants. CIS makes no representations, warranties or covenants whatsoever as to (i) the positive or negative effect of the Products or the Recommendations on the operation or the security of any particular network, computer system, network device, software, hardware, or any component of any of the foregoing or (ii) the accuracy, reliability, timeliness or completeness of any Product or Recommendation. CIS is providing the Products and the Recommendations “as is” and “as available” without representations, warranties or covenants of any kind. User agreements. By using the Products and/or the Recommendations, I and/or my organization (“we”) agree and acknowledge that: 1. No network, system, device, hardware, software or component can be made fully secure; 2.
    [Show full text]
  • A Guide for Employees (End Users) Last Updated: 7/10/2020
    Microsoft Home Use Program A guide for employees (end users) Last updated: 7/10/2020 Contents What is the Microsoft Home Use Program? .............................................................................................................. 2 Check if you’re eligible and sign up ............................................................................................................................ 2 Viewing products and offers ...................................................................................................................................... 4 Purchasing in the Microsoft Store .............................................................................................................................. 4 Redeeming and managing account ............................................................................................................................ 5 Download, install and activation ................................................................................................................................ 7 Frequently asked questions ....................................................................................................................................... 9 Employee Support Resources .................................................................................................................................. 10 1 What is the Microsoft Home Use Program? The Home Use Program (HUP) is an employee purchase program for employees of Microsoft’s valued customers. It offers Microsoft 365 Family
    [Show full text]
  • Owner's Manual Invoke
    INVOKE OWNER’S MANUAL1 INTRODUCTION Congratulations on purchasing your Harman Kardon Invoke with Cortana! For over 50 years, Harman Kardon has led innovation that enhances our customers’ audio experiences—a tradition we continue to uphold with INTRODUCTION the Invoke. In partnership with Microsoft, we are proud to bring this new experience with Cortana into your home. To fully enjoy your Invoke, please read this manual and refer back to it as you become more familiar with the product’s features and functions. Ifyouhaveanyquestions,pleasecontactyourHarmanKardonretaileror custom installer, or visit our website at harmankardon.com/invoke Sept 2017 2 ABOUT CORTANA Cortana® is your very own personal digital assistant. She’s always learning, so the more you askhertodo,thebettershe’llbeabletohelpyouwiththethingsthatmattermost. Whenever you need something, just say, “Hey Cortana,” and then ask your question. The ABOUT CORTANA topofthedevicewilllightuptoindicatethatCortanaheardyou. SomeofthethingsCortanacanhelpyouwith: • Playing your music, using the service you prefer • Callingfriends,family,andlocalbusinesses • Answering questions that come to mind • Helping you stay on track by managing your calendar, shopping lists, and reminders • Lighteningthemoodwithfacts,jokes,chitchat,andgames ForafulllistofeverythingCortanacando,checktheCortanaapponyourphoneorPC,or gototheCortanaskillswebsite:microsoft.com/en-us/windows/cortana/cortana-skills/ 3 PRIVACY TheInvokecontainsmicrophonesthatareactiveatalltimes,butonlytolistenforthewake- words,“HeyCortana.”Unlessyou’vesaidthosewordsandareinteractingwithCortana,no
    [Show full text]
  • Keep Gettig Notices About Microsoft Account
    Keep Gettig Notices About Microsoft Account Abraham is immunosuppressive and kotows semplice while corrective Benn wishes and warms. Unifilar and antiscorbutic Jimmie dabbles someso confer gal jurally.that Antone confided his accompanists. Gentianaceous and rakish Vassily dispersing her calentures hark while Aram stroke The problem goes away if I unlink the google account, Welcome to the Apple Support Communities. Thanks for assistance bad or use shared experiences, keep gettig notices about microsoft account you want i control my phone at microsoft, it installs malware removal process all ports through your. Home users and making me my code has been solved by any and keep gettig notices about microsoft account for help as a file. Software problems and keep gettig notices about microsoft account and then ok below manage files and how you. Removing their email messages to disable the quarantine page and keep gettig notices about microsoft account! Please use an email added to exercise your online mode to keep gettig notices about microsoft account is is over a microsoft accounts, you have to login to increase, a productivity nudge? That below to disable the context menu by changing the name suggests, and over a notification about? What give the difference between Gmail and MS Outlook. Since anything i keep gettig notices about microsoft account. If you keep gettig notices about microsoft account. If you are taught how small errors are increasingly going to keep gettig notices about microsoft account can follow us realizing it comes to longer exists first. Here to keep gettig notices about microsoft account? Total BS None of these suggestions work.
    [Show full text]
  • A Guide to Using Vbscript in Securecrt
    User's Guide Scripting Essentials: A Guide to Using VBScript in SecureCRT 4848 tramway ridge dr. ne suite 101 albuquerque, nm 87111 505 - 332 - 5700 www.vandyke.com A Guide to Using VBScript in SecureCRT Introduction 1 How This Document is Organized 1 Conventions Used in This Document 1 SecureCRT’s Scripting Objects/API Reference 2 Chapter 1: Creating Scripts 4 1.1 Starting from Scratch with a New Script 4 1.2 Recording a Script 5 1.3 Modifying an Existing Example Script 9 Chapter 2: Launching Scripts 11 2.1 Manual Script Execution 11 2.2 Automated Script Execution 20 Chapter 3: Connecting to Remote Machines 24 3.1 Connecting with a Pre-configured Session 24 3.2 Connecting in an “Ad Hoc” Fashion 25 3.3 Connecting in Tabs 26 Solution: Open a Group of Sessions in Tabs 28 3.4 Disconnecting Active Connections 29 Solution: Automatically Close an Inactive Connection (Auto Disconnect) 30 3.5 Connecting to a List of Remote Machines Within a Loop 31 3.6 Handling Connection Failures Within a Script 31 Chapter 4: Reading Data from Remote Machines 35 4.1 Accessing Selected Text on the Screen 35 Solution: Performing a Web Search with Selected Text 35 4.2 Waiting for Specific Data to Arrive 36 Solution: Receive Notification when “Error-Indicating” Text Appears 38 4.3 Capturing Data from a Remote Machine 46 Chapter 5: Sending Data to Remote Machines 54 5.1 Sending Plain Text 54 Solution: Repeat a Command with Variable Input from User 54 5.2 Sending Control Codes 56 5.3 Simulating Keyboard Events 57 Solution: Add "no" to Each Selected Line and Send to Remote 59 Chapter 6: Getting Information from the End User 62 6.1 Prompting for Simple Responses: Yes, No, OK, Cancel, etc.
    [Show full text]
  • Video Editing with Microsoft Movie Maker Contents
    Video Editing with Microsoft Movie Maker Contents Introduction ..................................................................................................................... 2 Creating a new project ..................................................................................................... 2 Importing footage ............................................................................................................ 3 Import from DV tape ..................................................................................................... 3 Question ................................................................................................................... 4 Getting started ................................................................................................................. 4 Editor view ....................................................................................................................... 5 Editing a clip ..................................................................................................................... 6 Splitting clips .................................................................................................................... 6 Splitting a clip ............................................................................................................... 7 Question ................................................................................................................... 7 Trimming clips .................................................................................................................
    [Show full text]
  • Windows 10: a New Generation of Windows Windows 10 to Be Offered As Free Upgrade; Group Computing and Holographic Windows 10 Devices Unveiled
    For Immediate Release Windows 10: A New Generation of Windows Windows 10 to be offered as free upgrade; group computing and holographic Windows 10 devices unveiled Hong Kong, 22 January 2015 – Microsoft today unveiled a new generation of Windows, with a wide range of experiences designed to usher in a new era of more personal computing, as well as two new devices designed to extend the Windows experience from large screens to no screens. Windows 10 will be delivered as a service to offer a safer, innovative and updated experience for the supported lifetime of the device. A free upgrade1 for Windows 10 will be made available to customers running Windows 7, Windows 8.1 and Windows Phone 8.1, who upgrade in the first year. “Windows 10 marks the beginning of the more personal computing era in the mobile-first, cloud- first world,” said Satya Nadella, CEO of Microsoft. “Our ambition is for the 1.5 billion people who are using Windows today to fall in love with Window 10 and for billions more to decide to make Windows home.” Windows 10: Making computing more personal Windows 10 marks the beginning of the more personal computing era, where technology disappears and people are at the center. In this era, the mobility of the experience matters, not the mobility of the device, and experiences should work in a seamless, familiar way across devices. People should be able to naturally interact with technology as they interact with other people — with voice, gestures and gaze. Protecting privacy plays a pivotal role in delivering trustworthy experiences that put people in control of their experience.
    [Show full text]
  • What Is Windows Vista?
    05_050918 ch01.qxp 6/14/06 8:18 PM Page 9 Chapter 1 What Is Windows Vista? In This Chapter ᮣ Getting to know Windows Vista ᮣ Introducing the new features in Windows Vista hances are, you’ve probably heard about Windows: the Cboxes and windows and mouse pointer that greet you whenever you turn on your computer. In fact, millions of people all over the world are puzzling over it as you read this book. Almost every new computer sold today comes with a copy of Windows preinstalled — cheerfully greeting you when first turned on. This chapter helps you understand why Windows lives inside your computer and introduces Microsoft’s latest Windows version, called Windows Vista. What Is Windows Vista, and Why Are You Using It? Created and sold by a company called Microsoft, Windows isn’t like your usual software that lets you write term papers orCOPYRIGHTED send angry e-mails to mail-order MATERIAL companies. No, Windows is an operating system, meaning it controls the way you work with your computer. It’s been around for more than 20 years, and the latest whiz-bang version is called Windows Vista. Windows gets its name from all the cute little windows it places on your monitor. Each window shows information, such as a picture, a program that you’re running, or a baffling 05_050918 ch01.qxp 6/14/06 8:18 PM Page 10 10 Part I: Windows Vista Stuff Everybody Thinks You Already Know technical reprimand. You can put several windows on-screen at the same time and jump from window to window, visiting different programs.
    [Show full text]
  • Your Microsoft Account & Services
    MICROSOFT ACCOUNT AND MICROSOFT SERVICES Why might I want a Microsoft account, what does it do for me? MICROSOFT ACCOUNT ALLOWS USERS TO SIGN INTO WEBSITES THAT SUPPORT THIS SERVICE USING A SINGLE SET OF CREDENTIALS. USERS' CREDENTIALS ARE NOT CHECKED BY MICROSOFT ACCOUNT-ENABLED WEBSITES, BUT BY A MICROSOFT ACCOUNT AUTHENTICATION SERVER • When you get a new Windows 10 device or install Windows 10 on an existing device the first thing the install or setup process will ask you for is an email address – that becomes your Microsoft account. It can be any email address. • The other option is to use a local account (this is the way the ASA PCs are setup) but Microsoft is making that more difficult to chose. There are advantages to using a Microsoft account. What are they? Your Microsoft Account ■ Access your Microsoft products and services with just one login. From Office and Windows to Xbox and Skype, one username and password connects you to your files, photos, people, and content. ■ To access your MS account and your information go to https://account.microsoft.com/ From here find all the devices linked to your account and their status. Find out what you have been doing and what Microsoft knows about you from tracking your activity. Ensure only the information you care about and find useful is there. How does having a MS account help me? Tools and resources Outlook ■ Access your email, contacts, calendar, attachments and to-do lists from one place ■ Outlook mobile app for iOS and Android Skype - https://www.skype.com/en/ OneDrive ■ Safely store and access your files and photos on all your devices.
    [Show full text]
  • Windows Movie Maker Introduction and Tutorial
    Last modified: 8/8/07 Windows Movie Maker Introduction and Tutorial Windows Movie Maker is free video-editing software included with Microsoft Windows XP. With Movie Maker, you can turn footage from your digital video camera into movies, share them on the Internet, and burn them to CD or DVD. Downloadable movies from online sources such as Unitedstreaming can also be edited using this software. Meeting the Maryland Teacher & Student Technology Standards The state of Maryland recently drafted and passed technology standards for students and teachers. For detailed information regarding the standards, you can view the teacher standards ( http://www.mttsonline.org/standards/ ) online or get a printable copy (http://www.mttsonline.org/standards/MDTchrTechStdsMTTS.pdf ) of the standards and indicators. Here is a link for the student technology standards (http://www.mcps.k12.md.us/departments/techlit/ ). • Maryland Teacher Technology Standards: o Standard 2- Communication : A. Use technology effectively and appropriately to interact electronically. B. Use technology to communicate information in a variety of formats. o Standard 3- Legal, Social and Ethical Issues : Demonstrate an understanding of the legal, social and ethical issues related to technology use. o Standard 4- Assessment for Administration and Instruction : Use technology to analyze problems and develop data-driven solutions for instructional and school improvement. o Standard 5- Integrating Technology into the Curriculum and Instruction : Design, implement and assess learning experiences that incorporate use of technology in a curriculum-related instructional activity to support understanding, inquiry, problem solving, communication and/or collaboration. o Standard 6- Assistive Technology : Understand human, equity and developmental issues surrounding the use of assistive technology to enhance student learning performance and apply that understanding to practice.
    [Show full text]