CALIFORNIA STATE UNIVERSITY, NORTHRIDGE Utilizing Microsoft Office Extensibility Within the Classroom Environment a Thesis Submi

CALIFORNIA STATE UNIVERSITY, NORTHRIDGE

Utilizing Microsoft Office Extensibility within the Classroom Environment

A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Software Engineering

By Yousef Haddad

December 2018

The thesis of Yousef Haddad is approved:

______Dr. Rick Covington Date

______Dr. Robert McIlhenny Date

______Dr. Taehyung “George” Wang, Chair Date

California State University, Northridge

TABLE OF CONTENTS Signature Page ...... ii List of Figures ...... iv List of Tables ...... v Abstract ...... vi Chapter 1: Introduction ...... 1 Chapter 2: Background on Microsoft Office Extensibility ...... 4 2.1 Common Object Model (COM) Add-Ins ...... 4 2.2 Visual Studio Tools for Office (VSTO) Add-Ins ...... 5 2.3 Office Add-ins and The JavaScript API for Office ...... 6 2.3.1 Office JavaScript API Add-in Architecture ...... 6 2.3.2 The Office.js library ...... 8 Chapter 3: Learning Management System Interoperability ...... 12 3.1 Sharable Content Object Reference Model (SCORM) ...... 12 3.2 Experience API (xAPI) ...... 14 3.3 Learning Tools Interoperability (LTI) ...... 15 3.4 Instructure Canvas ...... 16 Chapter 4: Design and Requirements ...... 17 4.1 Methodology ...... 17 4.2 Stakeholder Interviews ...... 19 4.3 Requirements and User Stories ...... 22 Chapter 5: Technical Implementation ...... 24 5.1 Used Technologies ...... 24 5.2 System Architecture ...... 26 5.3 Use of Instructure Canvas ...... 26 5.4 Role of the Back-End ...... 27 5.5 Authentication Flow with Canvas ...... 30 5.6 Interfacing with Office components ...... 34 Chapter 6: Recommendations ...... 37 References ...... 38 Appendix A: Interview Transcript 1 ...... 41 Appendix B: Interview Transcript 2 ...... 46 Appendix C: Interview Transcript 3 ...... 49

iii

LIST OF FIGURES

Figure 1: COM Add-in management within Word 2016 ...... 4 Figure 2: Architecture of VSTO Add-ins ...... 5 Figure 3: A task pane Add-in ...... 6 Figure 4: Generic Architecture for Office Add-ins ...... 6 Figure 5: Office Add-ins Store ...... 7 Figure 6: Add-in Manifest Validation ...... 7 Figure 7: Common API function example ...... 9 Figure 8: Use of context.sync() with the 2016 design of Office JavaScript API ...... 10 Figure 9: LinkedIn post calling for interview participants...... 19 Figure 10: Abstract architecture for QuizWord ...... 26 Figure 11: Data Fetch from Canvas Flow ...... 29 Figure 12: Canvas authentication ...... 31 Figure 13: Authentication Flow with Canvas ...... 33 Figure 14: Example of a course loaded in a content Control ...... 34 Figure 15: Flow of Content Upload to Canvas ...... 36

LIST OF TABLES

Table 1: Common questions asked to all interviewees ...... 20 Table 2: Interview Questions Specific to Professors and Learning Development Specialists ...... 20 Table 3: Interview Questions Specific to Students ...... 20 Table 4: Interview Questions Specific to LMS Implementation Specialists ...... 21 Table 5: Interview Findings ...... 21 Table 6: User Stories ...... 22 Table 7: Used Frameworks and Libraries ...... 24 Table 8: Used Cloud Resources ...... 25

ABSTRACT

Utilizing Microsoft Office Extensibility within the Classroom Environment

By Yousef Haddad Master of Science in Software Engineering

As learning management systems (LMS) continue to offer modern innovative features in educational delivery, educators and learners remain faced with a necessity of using productivity applications such as Microsoft Word, Excel, and Google Docs; often causing a discord in the end-to-end user experience while authoring content for submission. The need to export or upload externally managed files into the LMS platform generates swivel seat scenarios of switching between multiple programs. Office.JS is a Microsoft Office extensibility library that allows developers to manipulate content within a Word document,

Excel spreadsheet, PowerPoint presentation, etc. using JavaScript based application programming interfaces (API), while interacting with backend components, enabling end- users interaction with a task pane native to a Microsoft Office program. The capabilities of this library were studied through a proof of concept application that runs within Microsoft

Word 2016 (for PC, Mac), labelled QuizWord. The add-in facilitates interaction with

course content, assignments, and quizzes hosted within an Instructure Canvas instance for teachers to edit and students to consume while using Microsoft Word. The methodology involved conducting interviews with students and educators for requirement elicitation.

The project successfully enables content creation and consumption by teachers and students directly from Microsoft Word, achieving the goal of LMS interaction outside of a web browser, with an average response time under 1.5 seconds per information exchange round trip between a word document body and Canvas. Given the asynchronous nature of office.js, further optimizations can be implemented to improve performance. The variance between Hypertext Markup (HTML) and Office Open Extensible Markup (OOXML) was the main challenge that remains as room for improvement for this project to deliver a more accurate representation of document content into a Canvas’s standard HTML views.

vii

CHAPTER 1: INTRODUCTION

The use of computers is becoming a more integral part of education of all levels, including

K-12, higher education, and corporate training. Educators, students, technologists, domain experts, and other stakeholders, recognize the need for continuous evolution, and sponsored innovation within channels of electronic content delivery [1].

Learning Management Systems (LMS) are software applications derived to fulfill and facilitate concepts of e-Learning with multimedia and electronic based education, providing a platform for administration, documentation, tracking, reporting, and content delivery of educational courses or training programs [2].

Since its emergence in its current form within higher education in the late 1990s [3], the

LMS market continues to grow in value; the research organization MarketsandMarkets projects growth to $15.72 billion in worldwide revenue by 2021 [4]. This figure is driven by enterprises staying current by migrating from legacy LMS platforms or augmenting existing systems with interoperable tools to add functionality.

Outside of the corporate environment, reports show that 99% of colleges and universities worldwide use some implementation of an LMS platform [5]. Within the K-12 classrooms, initiatives are being envisioned and implemented on a wide scale on multiple administrative levels; in several US school districts, selection of LMS platforms or aiding tools is left to the educators’ discretion [6], regional school districts are implementing large scale initiatives in device distribution to students; in 2013, the Los Angeles Unified School

District has invested what was estimated to exceed $1 billion into providing iPad devices to 650,000 students, populated with Pearson’s Common Core System of courses [7]. On a

national level, the ConnectEd initiative, sponsored by the White House in 2013, aimed towards connecting 99% of K-12 classrooms in the United States with wireless Internet by

2018 at the cost of $3.2B [8].

While less Windows based devices were shipped in 2017, comprising 21.6% of that year’s market share, compared to the Google based Chromebook at 59% [9], Microsoft Office

365, the Software as a Service solution that includes Microsoft Office programs, has surged in enterprise adoption. A study conducted by Bitglass has indicated that Microsoft Office

365 deployments has increased in market share from 43% in 2016 to 65% in 2018 within the Europe/Middle East region (EMEA) [10]. This was attributed to heavily regulated finance and healthcare sectors, as well as the introduction of the General Data Protection

Regulation (GDPR) in May 2018, necessitating strict adherence to compliance standards for businesses with any customers based within the European Union [11].

The importance and prevalence of Microsoft Office as a productivity software suite are evident in the fact that Microsoft Office skills was considered as the third most sought- after skill by employers after oral and written communication skills and being detailed oriented; according to a study conducted by the International Data Corporation [12].

Having established the importance of both LMS platforms as well as Microsoft Office as a productivity suite with a rich set of tools for editing content, through this document, you will find the details surrounding the research conducted within this thesis; the Microsoft

Office extensibility library was used alongside an instance of Instructure Canvas to create a proof of concept that helps address swivel seat scenarios; or user experience interruption during the use of productivity programs in tandem with an LMS platform.

The research included conducting interviews with stakeholders, and part of the feedback received included the burden of managing individual files for assignments to be uploaded through the LMS user interface by the student. Educator participants pointed out not being able to integrate assignments and quizzes previously typed in a Microsoft Word document to the LMS platform being used without the need of reiterating data entry into specific fields on a web browser. Another pain point that was brought to light was the activity of exporting files and reports from an LMS platform into individual files through a web browser for further analysis was relatively cumbersome.

Over the rest of this thesis document, Chapter 2 will encompass the history and evolution of Microsoft Office extensibility features. Chapter 3 will discuss the topic of LMS interoperability and will introduce and differentiate multiple compatibility standards currently in use. Chapter 4 will discuss the methodology followed during designing the system, including requirement elicitation, user stories, and features. Details about the technical implementation and architecture will be the topic of Chapter 5. Chapter 6 will include findings and recommendations reached after the research was performed.

CHAPTER 2: BACKGROUND ON MICROSOFT OFFICE EXTENSIBILITY

Since the initial release of Microsoft Office in 1990, the popular productivity suite has undergone many changes in capabilities and features. Over time, platform support has grown to include PC and Mac, and now even includes iOS and Android mobile platforms.

Lightweight versions of Microsoft Office have been available through a web browser experience since 2010 through Office Online [13]. Among the features and improvements that have been added to Microsoft Office is the ability for users and third-party providers to create extensions to add further capabilities to such programs, such as specialized features and custom commands [14]. This chapter will discuss the timeline of Microsoft

Office extensibility features.

2.1 Common Object Model (COM) Add-Ins

Common Object Model or COM is a binary interface standard used for inter- process communication within Microsoft

Windows. COM was first introduced by

Microsoft in 1993 and has been available Figure 1: COM Add-in management within Word 2016 in Microsoft Office 2000 or later to extend functionality by allowing components to programmatically connect with other processes running within the host operating system, including third party applications. COM add-ins can be written in C++ and must implement the IDTExtensibility21 interface. They form the basis for many features that are perceived

1 IDTExtensibility2 is a .NET interface class that hosts event notifications between components

to be built into command bars, menu items, and inter-component communication [15], such as integration between Microsoft Word and other Office components, such as Microsoft

OneNote and Outlook.

2.2 Visual Studio Tools for Office (VSTO) Add-Ins

Visual Studio Tools for Office or VSTO is a set of development tools that allows Microsoft

Office 2003 or later applications to expose functionality via the .NET Framework Common

Language Runtime (CLI); allowing developers flexibility in choosing a .NET compliant language of their choice. Similar to COM Add-ins, VSTO add-ins are detected by an Office Application through registry keys in Microsoft Windows, and are run through event calls passing through an interop assembly that facilitates communication Figure 2: Architecture of VSTO Add-ins with the Office application’s COM-based object model. VSTO Add-ins communicate with

Office as COM Add-ins through COM wrappers; isolating IDTExtensibility2 implementation and simplifying developing solutions using only .NET libraries instead.

Figure 2 provides a simplified representation for VSTO Add-in architecture as displayed on Microsoft’s documentation website2.

2 Source: https://docs.microsoft.com/en-us/visualstudio/vsto/architecture-of-vsto-add-ins, last accessed Nov 20th, 2018

2.3 Office Add-ins and The JavaScript API for Office

The Office Add-ins platform allows building solutions using web technologies such as

HTML, CSS, and JavaScript that extend Office applications and interact with their content.

This can be done by referencing the office.js library within the web application.

Add-ins using the Office.js library run in the task pane space within the host Microsoft Office application. They are also available though customized command bars, as well as web dialog boxes, and provide users the ability to interact with an

HTML based GUI that can read and manipulate document content. Figure 3: A task pane Add-in

Through this section, we will discuss the general architecture of Office Add-ins that use office.js, its different versions, and what differentiates them from COM or VSTO add-ins.

2.3.1 Office JavaScript API Add-in Architecture

The Office.js library provides APIs that interface with Office application components and presents them as objects that can be used within a JavaScript application to achieve a goal. This allows developers to create web-based applications using familiar web technologies interfacing with a back-end infrastructure that extends functionality outside of the host device, providing endless possibilities to what users can perform, and how to visually interface with add-ins. Figure 4: Generic Architecture for Office Add-ins

Users can install add-ins through the Office Add-in store, which is a built-in component within Microsoft Office applications 2013 or later. Unlike COM and VSTO add-ins,

JavaScript based add-ins are compatible across Windows, Figure 5: Office Add-ins Store Mac, iOS, and the browser-based Office Online versions of Microsoft Office.

Office add-ins are referenced within its host Office application through a single XML manifest file, which indicates the Internet location where the web application is hosted.

Task Panes and Web Dialogs display the user interface through an Internet Explorer 11 web browser control, simplifying deployment and installation of add-ins by reducing the footprint taken to a single text file. Add-ins can be part of mass deployments of Microsoft

Office within an enterprise environment. Users can also sideload XML manifest files that describe the add-in by referencing them within the Office application’s settings.

The Office Add-in manifest, being the only component stored locally within the user’s device, contains meta data about the add-in, such as the location of the web application and its assets, target Office applications (Word, Excel, etc.), as well as any task pane or command bar related configurations. The manifest file can be automatically generated using tools provided by the Office Extensibility team and can be validated for syntax errors using an official JavaScript Node Package Manager (NPM) module; office-addin- validator, as shown in figure 6 below.

npm install -g office-addin-validator validate-office-addin MANIFEST.XML

Figure 6: Add-in Manifest Validation

2.3.2 The Office.js library

Office.js is an open source library developed by the Microsoft Office Extensibility Team, and maintains official visibility through GitHub3, alongside supporting content.

The library contains APIs that that represent Office application components and contents as JavaScript objects. Initially, the API was common among all supported Office applications of version 2013 or later. This includes Word, Excel, PowerPoint, Project, and

Outlook. Currently known as Common API, all objects use the root class Office.

The function Office.initialize() is called when the runtime environment is loaded, and the

Add-in is ready to start interacting with the application and hosted document. The same function is used for all compatible Office applications.

Office.context.ui.displayDialogAsync is another example that displays a dialog to facilitate further web navigation. Office.context.document is the interface that is used by the Add-in to access document content within Word, Excel, PowerPoint, and Project; providing a range of properties and methods that are common to all these applications4 that are run asynchronously.

For example, Office.context.document.getSelectedDataAsync reads data contained in the current selection within the body of a document object, whether that is highlighted text in

Microsoft Word, or selected cells in Excel. Other methods within the interface can be application specific, such as getTaskAsync, which serves only Microsoft Project.

3 Available through https://github.com/OfficeDev/office-js, last accessed in Nov 20th, 2018

4 Outlook is an exception where its components are referenced through Office.context.mailox

It is important to note that all methods that use the API to interact with the host application or its document object are run asynchronously, and take a callback function as an argument used to pass a result value, similarly to calling a web API. Below is an example.

function showSelection() { Office.context.document.getSelectedDataAsync( "text", // coercionType {valueFormat: "unformatted", // valueFormat filterType: "all"}, // filterType

function (result) { // callback console.log(result.value); // Carry on execution of the function }); }

Figure 7: Common API function example

The function is run asynchronously, and once executed successfully, it will pass it’s result into the callback function which will carry on the execution. The context interface that has been used in the function above represents the runtime environment of the Add-in, and provides access to objects of the API.

The Common API also supports binding with document information, where an event would be triggered whenever content being watched is changed.

It is important to note that the roundtrip for an API call to travel from the Add-in runtime within the Internet Explorer 11 component to access data within the document and then back to the Add-in can be expensive [16]. Within the Office 2013 design, every function call operates as a standalone call being sent to the Office host application, while the Add- in itself is waiting for a notification or data to be returned. This is a performance concern addressed in the Office 2016 API model.

With Office 2016 applications, loading office.js would also load a host-specific library; specialized to the given application being used, whether it is Word, Excel, PowerPoint,

Outlook, Project, or OneNote. While all functions and methods that use the Office root class are still usable with Office 2016 or later, office.js provides the option to access components specific to each Office application using its own top-level class.

For example, Word.document.getSeletion() loads a highlighted body of text within a Microsoft

Word document. The change within the 2016 API design is that instead of running every command as its own standalone asynchronous call, functions are executed against a proxy representation of the runtime environment (context), where actions are run sequentially and documented within a repository of actions. At certain points, developers can request the

Add-in to apply all changes to the document body using context.sync().

Below is an example.

Word.run(function (context) { // Queuing the command to read currently selected text var range = context.document.getSelection(); // Queuing the command to read selected text as HTML var bodyHTML = range.getHtml(); // Synchronize the document state by executing the queued commands return context.sync().then(function () { console.log('Cleared the body contents.'); }); Figure 8: Use of context.sync() with the 2016 design of Office JavaScript API

Word.run is initially executed, which generates an abstract proxy object that can be used to run commands in sequence without needing to wait for asynchronous calls to be resolved.

The context object would keep a log of all actions that need to be taken, and then

synchronizes them to the actual document when context.sync is called. Note that the only asynchronous commands that are being called in the code above are Word.run and context.sync, thus reducing the total time spent during calls back and forth to the object itself.

It is recommended to reduce the number of times context.sync is called within one’s code.

The office.js library can simply be included within the HTML page through a Microsoft provided Content Delivery Network (CDN) address. Another way to create an office.js based Add-in is to use either Visual Studio, or to use the Add-in generator through Yeoman.

Both methods are described in the GitHub documentation, and will generate an Office Add- in app’s scaffolding made ready for use, alongside an XML manifest.

The Office extensibility team also recommended the use of Office UI Fabric for the user interface as it maintains some conformity with what the layout and visual assets used within the actual host application.

CHAPTER 3: LEARNING MANAGEMENT SYSTEM INTEROPERABILITY

Prior to the development of the Sharable Content Object Reference Model (SCORM), electronic learning heavily relied on the ability of educators to store and share educational content including documents, multimedia files, syllabi, as well as student record management manually through self-made folder structures. The costs of modifying such content that was distributed was prohibitive [17], which drove the purpose behind the development of a standardized approach to organize course information in a way that encourages effective use of learning technology. This chapter will discuss this topic and its major mile stones since the late 1990s.

3.1 Sharable Content Object Reference Model (SCORM)

SCORM was initially developed as a solution to a challenge faced by the US Department of Defense when it came to managing distributed learning content, due to the nature of multimedia files being stored in a manual process, rather than an automated or standardized direction.

While the first technical standard for learning courses was created by the Aviation Industry

Computer-Based Training Committee in 1988, the AICC was formed by Boeing, Airbus, and McDonnell Douglass to implement structure into the training materials and platforms used to train airline workers; SCORM was created in 1999 and offered an upgrade on AICC

[18].

Unlike AICC, SCORM offers means to track course progress using an LMS platform, and was driven by a Presidential Executive Order [19] in 1998 to create a task force on learning technology, which has resulted in an empowered Advanced Distributed Learning (ADL)

Initiative program that falls within the scope of Department of Defense, that has since contributed to the development of widely adopted standards such as SCORM, and later the

Tin Can xAPI. A survey of 150 corporate LMS users conducted by SoftwareAdvice.com in 2018 indicated that 62% of businesses use SCORM as a standard for training courses

[18].

Content creation is relatively easy to create with the prevalence of SCORM authoring tools such as iSpring, Adobe Captivate, and Articulate Storyline among others. SCORM courses are in a ZIP file format and contain course contents alongside a manifest file that describes contents of a given SCORM package. SCORM courses can be uploaded into a compliant

LMS platform for immediate access to course content. SCORM has undergone several improvements until it reached the version SCORM 2004, 4th edition in 2009.

While there is a considerable adoption rate of this standard in corporate training environments, SCORM shows a number of weaknesses; for example, content is traditionally Flash-based; Flash content is not mobile device-friendly, and does not deliver as consistent of an experience as HTML5 does, though several authoring tools can output

HTML5 based SCORM content.

The other issue is the limited number of metrics that can be tracked; while SCORM can track metrics such as course completion, or time spent on courses, as well as assessment scores; real-time content engagement monitoring was not built into it as a standard, since it was focused more on the shareability aspect.

3.2 Experience API (xAPI)

Experience API, xAPI, also known as Project Tin Can, was developed by ADL; the creators of SCORM, as a need for a more robust e-learning standard was recognized. xAPI was released in 2013 [20]. xAPI as created with the purpose of allowing tracking more information about the learning behavior of users than what SCORM has offered. Through SCORM, metrics being tracked were within an LMS environment, and were limited to grades, completion rates, and amount of time spent per course. With xAPI, it is possible to customize what can be tracked during the learning experience. xAPI can be added as another layer of functionality on top of an existing LMS that utilizes SCORM packages, or otherwise.

The Tin Can API provides a modern approach where user behavior and activity is logged into a Learning Record Store (LRS) in an adaptive Noun – Verb – Object form. For example, if a student completes a training assessment, a record labelled as ‘Joseph completed Assessment.’ These records are called Activity Statements. Types of activities can be customized to what fits the organization’s needs.

Leveraging analytical tools can convert data stored in an LRS into information, and then knowledge based on observed trends and patterns of usage that are highly customizable and would assist in identifying user patterns that lead to success.

According to the same study referenced in the previous section, only 15% of corporate

LMS users use xAPI for training courses [18], however xAPI is considered as the innovative direction of LMS standards [20].

3.3 Learning Tools Interoperability (LTI)

While SCORM maintains high adoption rates among corporate training clients, LTI is considered as its counterpart in the educational scope of LMS platforms. LTI is a standard that was created by the IMS Global Learning Consortium, for the purpose of linking LMS platforms with external services and tools that are to improve upon the learning experience.

According to a study by Stefanova et al. SCORM is best used in environments where students are communicating largely through a computer, where IMS Global based standards such as LTI and Common Cartridge are more suited for a collaborative learning experience with further involvement by the instructor [21].

IMS Global describes LTI as a standard that prescribes how learning applications and tools can connect with LMS platforms securely. IMS Global provides other specifications; including the Common Cartridge is a specification for packaging educational digital content, and the Question and Test Interoperability (QTI) specification for representation of assessment questions, content and results [22].

LTI essentially defines the learning platform or LMS as a Tool Consumer, while the external or third-party tool providing extended services such as educational content, or a learning experience, as a Tool Provider. Within the LTI standard, the tool provider would operate while embedded within the platform, or Tool Consumer, while adhering to the specification outlined. LMS platforms that are LTI compliant have become a staple within the educational sector, including Instruture Canvas, Moodle, and Blackboard.

3.4 Instructure Canvas

Having introduced a brief background about LMS interoperability standards, including SCORM, xAPI, and LTI; this section will focus Canvas, an LMS platform developed by Instructure. Through the work of the project accompanying this research, Canvas was chosen to be used as part of the proof of concept implementation at the back end as we test integration with Microsoft Office Add- ins.

Instruture, the company that develops Canvas, is reporting consistent growth within the Higher

Education area reporting 30% of year-over-year growth within the second quarter of 2018 with a revenue of $50.1 Million. Instructure’s financial reports for the mentioned quarter highlights that

Cornell University has switched to using Canvas for over 22,000 students, Arizona State

University, having been ranked for the last three years as the nation’s most innovative school for the last three years by U.S. News and World Report has also adopted Canvas for their student body of over 90,000 . Instructure is also looking towards increasing its corporate training market share with their Bridge product line that is geared for this purpose [23]. The Canvas platform is also widely robust and is LTI, SCORM, and xAPI compliant. Canvas also offers extensibility through their own set of APIs that allows using other front-ends to leverage the Canvas infrastructure, which was fitting for the purposes of this project [24].

Canvas is available as either an open source option, or as a managed option with cloud hosting provided by Canvas. For the purposes of this project, a trial version of the managed Canvas instance was obtained for testing purposes, which is preloaded with demo-based courses.

Further details on the technical aspects of using the Canvas API are described in Chapter 5.

CHAPTER 4: DESIGN AND REQUIREMENTS

This chapter will discuss the methodology and processes followed during the execution of this project, starting from how the problem was defined and scoped, and how requirements for the solution were defined and implemented, and how implementation was validated against the criteria for success.

4.1 Methodology

The project idea’s inception has stemmed from personal observations while using LMS platforms alongside productivity applications. Coupled with my introduction to the JavaScript based form factor of Office Add-ins, I found an area of opportunity for improving integration between such applications to a more seamless, managed experience.

The generic intention of improving the LMS platform experience through new channels for delivery was refined into the following problem statement:

The use of productivity applications by students and educators remains decoupled from integration with centralized LMS platforms. Resulting swivel seat scenarios of switching between programs lead to increased margins of error, reduce efficiency, and consume time while adapting binary files between entities. The educational experience can improve through integration within a single screen without compromising the use of powerful editors.

The project was approached with the intent of proposing a viable solution that leverages Microsoft

Office extensibility features that are introduced through the JavaScript based APIs released with

Microsoft Office 2013.

I started with a comprehensive research about the current status of integration of educational platforms and current work. Findings were challenged to identify pain points. For example, synchronizing course content into device local storage comes at a high cost on performance; as shown within Microsoft OneNote Class Notebook extension [25]. Other alternatives that employ

LTI based integration reside entirely within a browser-based experiences using Google Apps, as is the case with Schoology LMS [26]. Getting to learn about existing work, read positive and negative recounts of user experiences has allowed me to better brainstorm ideas, and find out ways to innovate within the scope of technology being investigated.

The next step was to elicit requirements from relevant stakeholders through interviews. What followed was formalizing a list of user stories that are applicable to the technology being used. This will be the topic of discussion in the upcoming sections of this chapter.

Another aspect of this project is to investigate the usability of Microsoft Office Add-ins within such context and test its boundaries. The course of executing this project has been a learning experience into the office.js library. My approach to this has included a deep dive into the online documentation and publications authored by the team behind this tool [16], attending a quarterly community call that is held from Redmond, WA [27], involving enthusiasts, developers, and program managers introducing upcoming features, use cases, and best practices.

Another aspect of the methodology followed within this project was to avoid reinventing the wheel; instead of building a back-end database that would simulate an LMS environment, a design decision was to aim for integration with an existing LMS platform, to test the integration with a third-party platform. Instructure Canvas was the choice in this regard, largely due to their prevalence in the

LMS market, and extensive API.

Development followed an agile methodology that included iterations, revisiting features and user stories based on blockers and time constraints. This project was a single-person effort, and thus did

not include any collaboration or teamwork from a technical perspective. While agile principles were followed, the course of development was highly adaptive and was loose in timing, spanning over a four-month period, with 2 week long sprints. A CI/CD (continuous integration / continuous deployment) pipeline was implemented for delivery on a single stage meant to be for development.

API testing was done automatically, however, UI and end-to-end testing was performed manually for each user story.

The resulting proof of concept application was then benchmarked systematically by measuring response times of commands and measuring roundtrip requests to assess what kind of latency a user should anticipate should a similar solution be used, and documented through Azure

Application Insights.

4.2 Stakeholder Interviews

As mentioned in the previous section, current subject matter experts have been consulted for requirement elicitation purposes. Through this project, four interviews were conducted with the different types of stakeholders, with the intent of gathering further insight regarding use case scenarios, and pain points that need to be addressed, especially in relation to the use of multiple applications in a swivel seat scenario.

Through the interviews, we have engaged a graduate student, a college professor, a learning and development specialist from a corporate background, and an LMS implementation expert who works within a business capacity. Figure 9: LinkedIn post calling for interview participants The interviews were all conducted using Skype, using its recording feature. Each interview has lasted approximately forty minutes. Conversations comprised of open-ended questions followed by

discussions about more specific aspects of their experience in interfacing with LMS platforms.

Interviews have been transcribed and are available as part of the appendix of this document.

Interview participants have been invited to participate through LinkedIn; shown in figure 9.

Interviews included the following common open-ended questions:

Table 1: Common questions asked to all interviewees

Common Interview Questions 1 Can you introduce yourself and your role? 2 Can you describe the main technology used in your organization to manage the learning experience? 3 What other programs other than the central LMS platform do you typically use to get the job done? 4 Describe a typical scenario where you need to use productivity applications alongside and your LMS platform. 5 What would you change about your current experience in using computers and technology with education? 6 If you were in my shoes, what would you ask yourself, considering our discussion so far?

Other questions were asked for each specific role. For educators (professor and learning and development specialist), conversations began with the following questions:

Table 2: Interview Questions Specific to Professors and Learning Development Specialists

Interview Questions Specific to Professors and Learning Development Specialists 1 Describe your engagement with learners within your environment. 2 What other stakeholders are involved in the educational process? 3 How do you normally gather and analyze performance reports by your students? For the student participant, he was asked the following question:

Table 3: Interview Questions Specific to Students

Interview Questions Specific to Students 1 Describe your engagement with educators within your environment.

For the implementation specialist, the conversation has steered in a different, yet very rewarding direction, including:

Table 4: Interview Questions Specific to LMS Implementation Specialists

Interview Questions Specific to LMS Implementation Specialists 1 What type of stakeholders do you typically work with during implementation? 2 What are main asks from end users that you assist during training? 3 What are the main asks from management stakeholders?

Throughout the interviews, findings indicate the following:

Table 5: Interview Findings

Interview Findings 1 All participants use a central LMS platform (whether it is educational for academic participants, or instructional for training professionals. 2 Students and Teachers use a word editor (Google Apps and Microsoft Word) for drafting information prior to re-entry into the LMS platform. 3 Microsoft Excel was used by all educators for the purpose of refining and tweaking reporting data and information. 4 All users have expressed their frustration with the need to maintain parallel folder structures that stores deliverables, emulating what is displayed within the LMS platform. 5 Other programs used alongside an LMS platform by educators would be content authoring platforms. 6 A big pain point was the lack of a mobile friendly interface for the used LMS platform. 7 All participants have expressed that at least once, they have faced an issue with uploading a file to an LMS platform. 8 The implementation specialist pointed that while LMS platforms can push data into productivity applications, 2-way communication is very uncommon. 9 Feedback channels were generally more preferred to occur face to face, rather than through discussion channels on LMS platforms.

All participants have been duly thanked and followed up with. And are invited to test the Office

Add-in once a prototype is ready for use. General feedback regarding the concept of the add-in was welcomed.

4.3 Requirements and User Stories

While requirements were not directly elicited by interviewees or stakeholders, they were inferred from interview conversations, what current alternatives offer, and based on how functionalities offered by the office.js platform.

Table 6: User Stories

User Stories 1 As a user, I want to be able to log into my account securely. 2 As an educator, I want to be able to view a list of my taught courses. 3 As an educator, I want to be able to edit course content. 4 As an educator, I want to be able to use Microsoft Word to transform typed questions into questions with multiple choice, essay, or fill-in-blank answers available within an LMS platform. 5 As an educator, I want to be able to create and edit homework assignments. 6 As an educator, I want to be able to view a list of my students and their grades with a graphical representation. 7 As a user, I want to be able to keep using my existing LMS platform. 8 As a student, I want to be able to submit written assignments using Microsoft Word. 9 As a student, I want to be able to view a list of my enrolled courses. 10 As a student, I want to be able to view course content through Microsoft Word.

User stories listed above translate into the following features:

1. Integration to an existing LMS platform

2. Ability to view and edit course content

3. Ability to create and consume homework assignments

4. Ability to create quizzes and questions

Some of the asks that fell out of scope within this project include mobile device compatibility. While the Office add-in is compatible with iOS mobile devices, use of the document body will require a mouse and keyboard for a convenient experience. Another

notable exception is the ability to take quizzes from within Microsoft Office applications.

Since the Canvas LMS was being used as a back-end, the API to take and submit a quiz is not made available (an HTML web page would load in such instances).

CHAPTER 5: TECHNICAL IMPLEMENTATION

Part of this project was building a proof of concept Office Add-in that showcases the office.js library functionality, while interfacing with the Canvas API on the backend. This chapter will discuss technical implementation details behind the project. The application has been titled as QuizWord.

5.1 Used Technologies

The development of this application was performed mostly using the TypeScript. Below is a list of technologies and resources used to develop, deploy, and manage the project within its lifecycle.

The following frameworks and libraries were used to write the code of the application

Table 7: Used Frameworks and Libraries

Used Frameworks and Libraries 1 Angular 2: A JavaScript framework, used to build the front-end of the application. 2 Office.js (JavaScript API for Office): used to allow the front-end to interface with host Microsoft Office applications (Word). 3 Office UI Fabric: a JavaScript and CSS library used for styling the front end in a way that flows with the rest of the host application. 4 jQuery was used to supplement the design of certain components in the front-end. 5 Node.js was used to develop the back-end server, which handled all calls made by the front-end. 6 PowerShell: a command line language that was used to automate deployment in the Azure cloud.

Yeoman.io was used to generate the scaffolding for the template office.js front-end project, while Express.js was used to do the same for the backend.

Node Package Manager (NPM) was used to install and manage external dependencies, as well as execute runtime of the front end and the back end. Project compilation was performed using System.JS, Babel was used for transpiling the code from TypeScript into

ECMAScript 5 JavaScript so that it runs with polyfills as expected within an Internet

Explorer 11 environment.

Testing the application was done largely through Postman for the back-end APIs. Front- end testing and debugging within the task pane space was done through Microsoft Edge

Developer Tools.

The project was deployed in a cloud environment, hosted within Microsoft Azure. The following cloud resources were used to run the application:

Table 8: Used Cloud Resources

Used Cloud Resources Purpose 1 Virtual Machine (IaaS) Hosting the Angular 2 front-end application, development 2 Web App (PaaS) Hosting the Node.js back-end application 3 Azure Blob Storage (Object Storage) Storing image files 4 Azure Key Vault SSL Certificate and Secret key management 5 Azure DevOps CI/CD Pipeline and Repository 6 Azure Container Registry Container Registry

5.2 System Architecture

The application was deployed in a multitiered model. All requests that went through to the

Canvas API were relayed from the front end into a Node.JS back end that resided in the

Azure platform. The entire architecture

was deployed in Azure. California-state-university.acme.instructure.com

Azure KeyVault was used to store sensitive

information such as key secrets related to Container Registry Web App Weldergelder.azurecr.io Backend / Node.js qw-be.azurewebsites.net Storage blob qwstore.blob.core.windows.net the Canvas API as well as SSL keys to Azure Key Vault qw-keyvault.vault.azure.net certify connections over HTTPS, which is

Git repository a requirement for the Canvas API as well joehaddad.visualstudio.com Virtual machine Frontend / Angular.js qwfe.westus2.cloudapp.net Microsoft as using the office.js library. Figure 10 Azure shows an abstracted architecture diagram

Word that is used for the deployment. Figure 10: Abstract architecture for QuizWord

5.3 Use of Instructure Canvas

Instructure offers Canvas for use as either an open source version that can be managed by the user at no cost, or through a managed cloud instance hosted by Instructure. The cloud instance can be offered through a trial version for 2 weeks that can be extended. To accomplish this project, a trial version for a sandbox cloud instance of Canvas was used, which was conveniently preloaded with demo content and user accounts.

Canvas was used throughout the project to manage authentication, user management, and for data storage, except for images, which were stored in an Azure Blob container. The

main reason for this decision was that my own proposed design for an infrastructure of storage was very parallel to what was offered through Canvas, and in the same time, it would be a more relevant case study to showcase integration with an existing platform, rather than creating a new LMS and reinventing the wheel.

5.4 Role of the Back-End

The back end was used as an abstraction layer that facilitates communication between the front-end application hosted within Microsoft Word, and the Canvas platform. It performs the task of wrapping requests delivered from the front-end into compliant HTTP calls that are sent to Canvas. The back end will also standardize the responses from Canvas into a

‘success’ message, containing the requested result alongside a token, or am error message that gets reported from Canvas. There are two main benefits of using the back-end as it is.

The first benefit is to reduce the size of the front-end application. The add-in is running within a constrained environment as it is, and communicating with Canvas with its different

API endpoints will require vastly varying sets of logic to process the input and output.

While Canvas is open source and behavior is to be expected, there is a chance of introducing breaking changes, causing responses to be different in a way that requires further logic to deliver a user-friendly prompt in such scenarios. The back-end relieves the front-end from these tasks. The endpoints existing within the back-end were simplified so that they would include the following:

1. /authinstructure (POST): This endpoint takes an object containing an authorization

code, and sends back either an error message, or a success message with an access

token and a refresh token.

2. /getinstructure (POST): This endpoint takes an object containing the access token,

refresh token, and a string that signifies URI parameters that need to be included

within a Canvas API call. This way, all front-end components can share the same

HTTP post call to request information to be read, whether it is a list of courses,

modules, or contents of a page.

3. /updateinstructure (POST): This endpoint takes an object containing the access

token, refresh token, a string of the URI paramteres needed within the subsequent

Canvas API call, alongside a payload object containing the information needed to

be updated, or used to create a new item; such as a new page or assignment.

The other benefit of decoupling the front-end from Canvas with a back-end layer is to accommodate any future feature changes, such as user activity tracking, communication with other platforms, such as xAPI LRS store, and the addition of further features that can be combined with the Canvas experience.

It is also worthy to note that the back-end is responsible to keep the user authenticated in the case of access token expiry. If any call is rejected by Canvas due to an expired token, the back-end will automatically renew the token with Canvas, and then resend the required request before relaying the information back to the front-end. The next figure illustrates the flow of fetching data from Canvas through the back-end.

Data Fetch from Canvas Flow

Task Pane Add In Back-End Canvas Instance

Start

Send Request to Wrap request and Send response back Back End send to Canvas to Back End

Data sent back to add-in with used Yes Valid Result? Token

Authenticated Behavior No

Process Refresh Send Refresh Token Token and send to Canvas response

Token Refreshed?

Yes

Include new Token in wrapped request

No Valid Expired Token Behavior Expired Valid

Update/Delete Authentication current token from Failure Message local session with empty token

End

Figure 11: Data Fetch from Canvas Flow

5.5 Authentication Flow with Canvas

Canvas requires all calls to be authenticated. Canvas currently uses OAuth 2.0 for authentication purposes; every HTTP request made to canvas must have a valid access token in the header.

For testing purposes, Canvas offers the option to manually download an authentication token that is then stored within a browser’s local store. This requires a user to request an authentication token through a system administrator as a binary file. This approach is recommended only for developers intending to test the API without the need to adapt an application’s architecture to the authentication flow requirement by Canvas.

The use of Canvas’s authentication model also relieves the need to add further components to store, process, and manage user credentials. Canvas provides a clear structure for user levels with varying authorization levels that is inherently enforced within the application, without the need to apply intensive logic to validate user authorization levels externally.

User levels within Canvas includes admin, teacher, student, TA, observer, designer; of these levels, QuizWord caters to Students and Teachers.

Establishing the authentication flow within QuizWord was the first work item that was accomplished within this project; given its importance to streamline execution and testing of all other future features, and the relative complexity of the flow, noting that Canvas requires the implementation of a particular flow to complete authentication. Figure 12: Canvas authentication

As a first step, a Developer Key was created through the Canvas admin dashboard, through which an application ID and Secret are provided for use. A redirect URL must also be provided, which will be the next hop Canvas will redirect users to after authentication. The user experience will show an web page representation of the Canvas login interface before redirecting back to the application. The Web Dialog component of the Office JavaScript

API was utilized for this purpose, as shown in Figure 11.

Once authentication is complete, Canvas will redirect users within the Web Dialog component to the QuizWord application server, delivering the result of authentication, with an access code as a URI parameter, in the event of successful authentication. The front-end application will then forward the response to the back-end server for processing.

The application’s back-end is used for processing the token response in an effort to reduce client-side processing; this will include processing a received authorization code to obtain an access token that will in turn be sent back to the web dialog component, where the access token will be stored in a cookie.

This was also built with the intention of maintaining the scalability of the application; should any user logon data become needed for further processing outside of Canvas in upcoming features.

Tokens provided by Canvas have a one-hour long lifetime and can be renewed using a refresh token. QuizWord requires users to authenticate once and will handle re- authentication using the refresh token through every API call, if needed.

Figure 12 illustrates the flow of authentication between the front-end instance of

QuizWord, including the task pane and web dialog interface components, alongside with the cloud-based Canvas instance.

Authentication Flow with Canvas

Task Pane Add In Dialog Box Back End Canvas Instance

Start

Request Canvas Send Canvas Login Login URL Page User clicks to log in

User performs

authentication User Input User PhaseInput

Check Credentials

Remain Valid Call Redirect page No unauthenticated Credentials? with results

Use authorization Prepare and send Yes code to request Token and Refresh Token Token

Validate Token and Navigate to Course Store Token data in Refresh Token List page a cookie

response Authentication and Token Exchange Token and Authentication

End

Figure 13: Authentication Flow with Canvas

5.6 Interfacing with Office components

The application runs entirely within the scope of Microsoft Word. The application is run in a task pane format through a JavaScript based interface. The Web Dialog control shown in the Figure 11 is used to fulfill the authentication requirement by Canvas.

Content from the Canvas back end is fetched in HTML format, and is parsed and inserted into the body of a word document. Microsoft Word documents provide Content Controls that facilitate sectioning controlling the content through the API in a visible manner to both the user and the program. This has been used alongside partitioning the application’s runtime through sections.

A limitation of the Office.js environment is images support. Images are only supported within the

Office API if they appear inline alongside text, where floating images are not detected. In order to circumvent this issue, prior to saving any content, images and non-text content such as charts, handwritten ink, mathematical notations, etc. are converted in an inline format, users are shown a rendered version of their content to be altered prior to be pushed back to the server to ensure that graphical Figure 14: Example of a course loaded in a content Control content is not being snapped into undesired locations.

Microsoft Office maintains its content in a specific markup known as the Office Open XML markup

(OOXML). This allows documents within Microsoft Word, Google Docs, Apple Pages, and

OpenOffice to be compatible in showing content within documents, including charts, relative location of items within the document, styles, and other shapes. In our current scenario, content is

being interpolated into HTML markup rather than OOXML, which might be rich, however, does not have an immediate mapping to all OOXML tags and components.

Through this project, all text and table formatting is preserved, however, any other item other than text, such as a shape, handwritten note, math equation, etc., has been converted into a PNG image and stored within Azure Blob storage to be referenced within the HTML code on Canvas. Images are represented as binary objects within a Microsoft Word document. The base-64 representation of each image is accordingly streamed into the back-end prior to being uploaded to blob storage.

The location of each image is then updated accordingly within the HTML tag within each picture, so that it is displayed properly once viewed later. Figure 15 describes the flow of uploading data into Canvas within QuizWord.

Content Upload to Canvas Flow

Microsoft Word Task Pane Add In Back-End Blob Storage Canvas Instance Document

Start

Edits made within User enables edit the Word document mode space

OOXML converted to HTML in a new document section, User clicks on graphic elements Review Changes converted to inline pictures

HTML representation User confirms erased, document No changes? scrolls back to

OOXML data User Interaction Yes

Convert Base64 Stream graphical Send Base64 image notation into .PNG objects in Base64 to representation to files and store back-end Blob Container images

Add image Blob URL Process and forward to each picture s src image URI to front-

field end Handling Graphics Handling

Send access token, refresh token, and Wrap request Update / Create new Canvas URI according to Canvas content parameters, and API specification HTML payload

Reload with new Forward response to Yes Success? content Front-End

End Display error

Save content in in Save Canvas content Figure 15: Flow of Content Upload to Canvas

CHAPTER 6: RECOMMENDATIONS

Testing results for the projects showed an average response times of 1455 milliseconds per trip between the Word document body to Canvas. This was measured by benchmarking the functions as they run within the program, and then aggregating results within a table storage and Excel. It was found that operations involving Canvas communication over the Internet take an average of 620.88 ms, file uploads (average of 5 pictures per document) takes 246.3 ms, while exchanging information between the Add-in and the Word document took the longest period of time; 957.8 ms, which clearly shows that optimizing the Office.js usage within the program is key for performance improvement.

Another area to improve is the mapping between OOXML and HTML so that more document related resources and their location within a document is translated into HTML, rather than producing images that are snapped to an inline-with-text appearance. The third area for opportunity is adapting the program for compliance within legal standards such as the ADA, to validate its use within the educational industry, as well as the addition of further features to improve feasibility.

This comes as an improvement to the manual process of managing files individually and using upload tools to move data from Microsoft Word into Canvas. Another benefit was the preservation of formatting for word documents through HTML encoding. As next steps, this project will be moving forward towards a more stable release that will be distributed through the AppSource store for the public to use, involving more stakeholders into the requirement generation process.

REFERENCES

[1] Worthington, M., “3 Ways to Implement Ed-Tech to Help Our Students Succeed,” Education Week, February 2016.

[2] Ellis, R. K., “Field Guide to Learning Management,” ASTD Learning Circuits, 2009.

[3] Davis, B., Carmean, C., Wagner, E., “The Evolution of the LMS: From Management to Learning,” The eLearning Guild Research, Iss. 24, 2009.

[4] Jacobs, S., Cantu T., De Vries, J., Richards, D., “Overcoming Common Pain Points When Switching to a New LMS,” The eLearning Guild White Papers, February 2018, https://www.elearningguild.com/sponsored/1552/overcoming- common-pain-points-when-switching-to-a-new-lms/, last accessed Nov 20th, 2018

[5] Mehta A., Kalyvaki M., “Learning Management System: Education Research in the Era of Technology,” Ch. 2 pp 9-19, American Chemical Society, November 2017

[6] Molnar, M., “Wave of Open Content a Challenge,” Education Week: Special Report on Extending the Digital Reach, 2016.

[7] Herold, Benjamin, “L.A. iPad Program an Ongoing Mess, Evaluators Find,” Education Week, 2015.

[8] Mareco, D., “Obama’s Plan to Take Our School WiFi Networks into the 21st Century.” https://www.securedgenetworks.com/blog/obamas-big-plans-for-your- school-wireless-network-in-2015, last accessed Nov 20th, 2018

[9] Finn, N., “Global K-12 Mobile PC Market Projected to Pick Up in 2018, as Large National Projects Compensate the Slow Down in the USA: K-12 Mobile PC Market Research, Analysis and Commentary,” Futuresource Consulting, https://futuresource-consulting.com/Press-K-12-Education-Market-Growth- Forecast-in-2018-0318.html, last accessed Nov 20th, 2018

[10] Targett, E., “Office 365 Gobbling Up G Suite Market Share,” Computer Business Review, 08/22/2018, https://www.cbronline.com/news/g-suite-vs- office-365, last accessed Nov 20th, 2018

[11] Das, A. C., “US Companies Still Grappling with GDPR,” The National Law Review, 08/22/2018, https://www.natlawreview.com/article/us-companies-still- grappling-gdpr, last accessed Nov 20th, 2018

[12] Anderson C., Gantz, J. F., “Skills Requirements for Tomorrow’s Best Jobs Helping Educators Provide Students with Skills and Tools They Need,” White Paper, IDC, October 2013

[13] Finley, K., “Microsoft Rolls Out Office Web Apps,” 06/08/2010, http://www.readwriteweb.com/enterprise/2010/06/microsoft-rolls-out-office- web.php, last accessed Nov 20th, 2018

[14] Bort J., “Microsoft Shows Off a New App Store for Office 2013,” https://www.businessinsider.com/microsoft-office-2013-app-store-2012-8, last accessed Nov 20th, 2018

[15] Turner, L. B., “Excel COM add-ins and Automation add-ins,” https://support.microsoft.com/en-us/help/291392/excel-com-add-ins-and- automation-add-ins, last accessed Nov 20th, 2018

[16] Zlatkovsky, M., “Building Office Add-ins using Office.js,” Lean Publishing, 2017

[17] Ostyn, C., “A brief introduction to SCORM,” click2learn SCORM 1.2 Resource Kit, https://scorm.com/wp- content/assets/cookbook/SCORM%201_2%20Overview.htm, last accessed Nov 20th, 2018

[18] Westfall, B., “ACORM vs. AICC vs. Tin Can (xAPI): Why xAPI is the Only Way Forward,” Software Advice, https://www.softwareadvice.com/resources/scorm-vs-aicc-vs-tin-can/, last accessed Nov 20th, 2018

[19] Clinton, W. J., “Using Technology to Improve Training Opportunities for Federal Government Employees,” Executive Order 13111, The White House, January 1999

[20] Winner, A., “An Introduction to Tin Can API,” eLearningIndustry.com, https://elearningindustry.com/introduction-tin-can-api, last accessed Nov 20th, 2018

[21] Stefanova, S., Spasov, S., Zdravev, Z., “SCORM vs Common Cartridge – Case Study at University Goce Delcev,” The Fifth International Conference on e- Learning (eLearning-2014), September 2014

[22] IMS Global Learning Consortium, “Learning Tools Interoperability,” https://www.imsglobal.org/activity/learning-tools-interoperability, last accessed Nov 20th, 2018

[23] Instructure, “Instructure Reports Second Quarter 2018 Financial Results,” Jul, 2018, https://www.prnewswire.com/news-releases/instructure-reports-second- quarter-2018-financial-results-300688104.html, last accessed Nov 20th, 2018

[24] Instructure, “Canvas LMS API Documentation,” https://canvas.instructure.com/doc/api/index.html, last accessed Nov 20th, 2018

[25] Powell, P., “How to Use Class Notebook Efficiently,” Vancouver School Board, https://office365.vsb.bc.ca/how-to-use-class-notebook-effectively, last accessed Nov 20th, 2018

[26] Ravipati, S., “Schoology LMS Enhances Google Drive Integration,” The Journal, June 2017, https://thejournal.com/articles/2017/04/06/schoology-lms-enhances- google-drive-integration.aspx, last accessed Nov 20th, 2018

[27] Microsoft Office 354 Blog, “Office Add-ins community call-October 10, 2018,” https://developer.microsoft.com/en-us/office/blogs/office-add-ins-community- call-october-10-2018%E2%80%AF/, last accessed Nov 20th, 2018

APPENDIX A: INTERVIEW TRANSCRIPT 1

Contact: Abdullah Shaban Role: Product Manager at Petco, current MBA candidate at the university of San Diego Location: San Diego, CA, USA Topic: Use of LMS platforms alongside productivity apps at University of San Diego Date: May 14th, 2018

YOUSEF: Hello and welcome, Abdullah Shaban, I appreciate you taking the time today for a brief interview. Let’s get started, shall we? ABDULLAH: Let’s go. YOUSEF: I know we spoke about this briefly prior to the call. This is really interviewing you as a student, as a higher education student who is pursuing a graduate degree, who interfaces with a lot of systems provided by the school, including Learning Management Systems. Our project specifically targets swivel seat scenarios; which are instances where you need to use multiple systems, switch back and forth at any capacity and reiterate data entry to achieve a certain task. We’ll get to this at a later stage. For now, can you please provide me with a brief introduction about yourself as a student? ABDULLAH: My name is Abdullah Shaban, I am currently a Product Manager at Petco; I am responsible for managing our supply chain systems in addition to the infrastructure for our e-commerce platform. I am also pursuing my MBA degree at the University of San Diego; it is a two year executive part-time program. I am almost done with it now. I am originally from Jordan, I have been in the US for about 2 years and a half. YOUSEF: Thank you. It sounds intense to have to lead the life of a full-time job and go to school altogether, especially with some of the MBA programs, even when they are part- time, they can be quite demanding. It’s quite admirable what you are doing. As a student, can you describe what the engagement is like between yourself and educators? ABDULLAH: I am in a relatively smaller school. One of their competitive advantages is that they claim they have much smaller classrooms, so they can provide a lot more of a personal touch to students with their educators. Based on my experience, this has been the case. Our largest classes would include 25-30 students; professors would know you by name and would be able to help you during or outside office times. They are often reachable by email. I found them extremely supportive, that if you ask the right questions, they are always there to support you. YOUSEF: What you’re telling me there is a lot of direct contact, or personal engagement between you and the educators in terms of seeking information and getting answers back,

having the direct feedback channel instead of having it automated or having it delivered through some mass delivery platform. ABDULLAH: It’s a mix of both; my professors sometimes need to communicate through Blackboard5, and use it to send mass announcements to the entire class regarding changes to the syllabus, preparation requirements, and so forth. We would still have one-on-one conversations, which we can establish by approaching them during office hours or after class, or just through emails. YOUSEF: Okay, so you mentioned emails and Blackboard. Would you describe Blackboard as the central or official LMS platform that your school uses? ABDULLAH: We mainly use Blackboard for everything related to course material, syllabus information, but we also use Google Apps in general for us to be able to create shared documents and drives for our teams to collaborate a little better. YOUSEF: When you mention collaboration using productivity applications such as Google Apps or otherwise, let’s take Google Docs for now as an example. What would this collaboration look like? ABDULLAH: It would be mainly for assignments that are between me and a group of other students. YOUSEF: In other words, you are using a word editor for collaboration. What is the integration between Google Docs and the central LMS provided by your school? ABDULLAH: There is no integration, what you would have to do is, after the assignment is over, you will need to download the document, reformat it so that the format isn’t broken after changing it from a Google Doc to a Word Doc, or Microsoft Word doc, and then you would need to submit it on Blackboard. YOUSEF: Okay, that all makes sense. Can you describe any other similar scenario where you need to switch back and forth between one application and another? ABDULLAH: So, I would say this is the case for most deliverables. For example, in more technical classes, we are required to create some financial models. Those would require us to create the content using Google Sheets, which is not as simple and easy to use as Excel, so you would use Excel to create the document, save the file and then upload it to Blackboard. The same would happen with PowerPoint presentations as well. It is all deliverables, really. YOUSEF: This basically sums it up for me, as a word; deliverables. It’s pretty well spoken too. Within that experience of needing to use one program and then the other, you mentioned needing to work a little bit more on formatting to make sure that content carries over normally, before being exported at some capacity to be viewed within the Learning Management System. How about from the other way around? What is some content that exists within the LMS platform that you would need to export, are there any scenarios that resemble this description?

5 Blackboard is a Learning Management System developed by Blackboard Inc.

ABDULLAH: For example, templates that we need to fill. Also, questions that we need to print out to be able to answer. That’s about it. Excel templates, I would say, are the biggest use case where we need to download them, fill our work, and then make sure that the information is filled in the right way, so that the professor can easily find out if we were able to get to the answer the right way. YOUSEF: Okay, so basically it is a boilerplate of sorts that you fill the gaps within before uploading back again. Among these deliverables you mention, whether you acquire them from the centralized LMS into your word editor back and forth, are they always files, such as .doc or .xls files? ABDULLAH: Yes, that is correct. YOUSEF: Within this entire experience, I noticed that you’re switching back and forth, and you described the experience quite well. Can you please identify some pain points, or experiences that you would rather see improved? What would you change if you had a magic wand? ABDULLAH: I think I enjoy using Google Apps just because they’re so easy and convenient. Collaboration is well built within Google Apps, it would be amazing to have group projects where everyone signs in, and once everyone is done, every participant would sign off on the final work to be submitted, and then afterwards, you can move the status of the document to be submitted through some automated workflow where the professor can check the work and leave comments that can be seen afterwards, all within the same space rather than needing to switch back and forth. Right now, you would need to submit the document as a file to Blackboard, Blackboard has a preview feature for comments which is not very intuitive and does not entail any notifications once a professor is finished grading an assignment. You would need to count on your professor to send a note on their own to be informed. YOUSEF: What you’re mentioning is keeping things centralized at the hub of where we start working on it, instead of having to toggle back and forth. ABDULLAH: Exactly. Each semester I start by creating a folder that is the semester folder, and then I create a subject folder within the semester folder, and break things down based on how the professor is showing information within Blackboard, just so that I can make sure that all my information is easily accessible based on how the professor decides to split up or organize the blackboard course. It becomes extremely tedious to be able to manage and make sure that all files you have in your Google Drive are the same as what you have in Blackboard. YOUSEF: This makes sense. I can see how this could be a frustrating experience, to have parallel folder structures that are similar, while you are maintaining your own copy. To recap what we discussed so far, you mentioned how you use Blackboard within your educational environment. It sounds for the most part like collaboration has been a big pain point that you have had to go through, especially when needing to use Google Apps, where you leverage collaboration and editing features, but you would still need to work further to refine the file before it is a complete deliverable that can be uploaded. The other piece is needing to match your own folder structure to what is on Blackboard so that you wouldn’t

have to lose traction; it’s not the most organic or intuitive experience and involves some level of effort to remain organized. Let’s switch to the topic of grades and score cards. How do you normally interface with this type of information? ABDULLAH: It’s available on Blackboard, all of it. Grades and comments for assignments are all on Blackboard. Exams, quizzes and all this information is also on Blackboard. YOUSEF: When you say grading, you also mentioned comments that are presented by your professor or educator, are these typically noted in-line within every question? Or would it be an overarching body of comments for an assignment that is submitted? ABDULLAH: It depends on the professor; I have seen it done both ways. Professors who are detail-oriented point out specific things, highlight them and place comments, where other professors provide overall comments about your work or submission. YOUSEF: Other than Blackboard, what other aspects that you use your within your computer for education, especially in a program that is not specific to a computer related topic, being business administration, do you have feedback about that you feel need to be addressed? ABDULLAH: Let me think. I think I have problems in taking notes when I’m reading case studies in PDF format. Especially that we get a lot of case studies where the professor would take a scanned copy of a case and send it to us as a PDF file. We would end up having to take notes on top of that. I have used multiple notes applications for me to make sure I have all my information digitally available I never really found a solution that works for me on all platforms that provides me with constant access to this information. I tried OneNote6, but I couldn’t access the information as easily as I want to on my phone. I have tried Evernote7 but felt it didn’t have enough features for me to use as a note taking application for all my educational needs. I think this is a big missing piece. YOUSEF: This is a very important topic you bring up; cross-platform compatibility. How would you like to view your notes on a bigger screen device such as a computer or laptop, or perhaps a tablet, versus viewing it on a small screen such as a phone? ABDULLAH: I think, seeing my information as if they are in a word document would be fine for a new topic or subject. If it is for an existing paper, such as a PDF file, I would rather see them overlaid on the existing document to understand the flow of the document; how I structure my notes would make a difference. YOUSEF: You mentioned that these are PDF handouts. Within your environment in the school, what is the percentage, or how often, do you see handouts being delivered as individual PDF files as opposed to content being published within the Blackboard environment? ABDULLAH: I would say it’s 70% PDF, 30% handouts.

6 Microsoft OneNote is a note taking application developed by Microsoft 7 Evernote is another popular note taking application developed by Evernote Corporation

YOUSEF: By handouts, do you mean information typed into the Blackboard system? Or what do you mean by that? ABDULLAH: I meant physical papers given in class that are not available within Blackboard. YOUSEF: So there’s two parts to this, when you think about the content delivery challenges, it is coming in a somewhat locked down format such as a PDF or an image scan, and the other part is not being able to place your comments in some application that provides a seamless cross-platform experience. Would you agree if this sums up what you were mentioning here? ABDULLAH: Absolutely. YOUSEF: Abdullah, I appreciate your time. You were very generous in giving me your insight regarding your experiences. Before we go on and wrap this up, I would like to ask you what you think I should have asked you to get more information about your experiences? ABDULLAH: To me, as a graduate student, that uses multiple platforms to complete their work, I think you asked questions that I can think of at this point. I think you’ve done a good job in expanding on my answers and making sure that I am responding to the right type of question. I don’t have any further recommendation of what you should be asking. I think you’ve done a good job in asking me that. YOUSEF: Thank you again for your time and for your kind words.

APPENDIX B: INTERVIEW TRANSCRIPT 2

Contact: Shayna Chan Role: Learning and Development Specialist at Microsoft Location: Los Angeles, CA, USA Topic: Use of LMS platforms within a corporate training environment Date: June 3rd, 2018

YOUSEF: Shayna Chan, thank you very much for taking the time to conduct the interview today. SHAYNA: Thank you, it is a pleasure of mine to be help. YOUSEF: Let’s get started. To give you a brief description about the project; it is about integrating Learning Management Systems into productivity apps such as Microsoft Word, Excel, and PowerPoint. Part of my research is to find out about challenges and pain points that educators and learning and development specialists such as yourself face while using such systems, which brings us to our interview today. First, can you please introduce yourself and your role? SHAYNA: I am Shayna Chan, I am a Learning and Development Specialist at Microsoft in Los Angeles. My role is to oversee learning and development for my team, which includes the titles Product Advisors, Service Advisors and members of local leadership teams. My responsibilities include delivering company initiatives to make sure the entire team is sharing a certain message as we interact with customers about products, as well as development, clarifying career progression, and making sure we utilize learning courses and actively engage what we learn in what we do. YOUSEF: That sounds exciting. It’s not just about having people learn about product knowledge, but also about paving the way for their career path. Would you describe yourself as an educator, with your students or learners being sales and service associates? SHAYNA: Yes. YOUSEF: How would you describe your engagement with your learner audience? SHAYNA: My role does resemble a teacher’s role. My focus is mainly assessing how employees learn best, employing specific teaching techniques for each member of the team, and prepare everyone to be an expert who can also teach others, which is a good way of learning. My role also includes testing team members’ knowledge. I also send out learning courses that must be completed by employees that are tracked in terms of completion rates, followed up with personal knowledge assessment to make sure that course material has been retained accordingly. It is not a classroom environment; however, it could be the case

in rare occasions. For product training, live demonstrations are often employed in training for a hands-on experience. YOUSEF: So, you’re saying that your teaching approach is truly different from one individual to another as everyone learns in a different way, and your engagement starts from providing learning courses, checking completion rates and following up with personal coaching or training to solidify what they learn, and test their knowledge. You also mentioned a practical aspect including live product demonstration. You mentioned tracking progress with courses. What is the main management system that you use for tracking this type of data? SHAYNA: We use a system called Learning Academy, which is a specialized version of ExpertZone8, which is a Microsoft owned platform. We have multiple options to track usage by content or course, or by user. Based on the completion metric that we are tracking, we observe employee completion percentages among team units, which helps us guide the effort towards identifying engagement within the team. It would also help provide an insight into what courses and topics individuals are most interested in. YOUSEF: Do you author any of the content within this system? SHAYNA: Content is created by another stakeholder and is distributed for use by from headquarters. I occasionally create content that gets published within the platform. YOUSEF: I see, so it is mostly furnished, with an option to provide insight and content to be added. SHAYNA: Exactly. YOUSEF: Apart from the centralized system, what other systems do you use to accomplish your tasks within your role, such as word documents, Excel sheets, or otherwise? SHAYNA: Yes, I do use OneNote to keep track of my engagement with team members, and create charts related to completion rates in a more customized way. It would also be a good place for me to keep key points and questions to ask to validate knowledge retention through readiness checks. I also use Microsoft Teams9 to communicate announcements with the team. YOUSEF: You mentioned you keep graphs and charts on some table that you take into one-on-one coaching sessions to address knowledge gaps. Do you feel that the learning management system you use provides the visualization to achieve that so that you wouldn’t need to re-enter information separately into another platform? SHAYNA: Within the system, we can export data, however, it would still need to be tweaked at some level, aggregated so that I can track what I need based on my objective. Another feature I need to emulate is the quiz feature. Most courses include some quiz that is required to be completed for tracking purposes, however, it does not accurately measure the level of understanding, especially that questions are largely multiple-choice questions

8 ExpertZone is now a learning management system used by Microsoft for retail training 9 Microsoft Teams is a team collaboration application developed by Microsoft

that can be taken in multiple attempts, and thus may not reflect employee knowledge. I would normally need to recreate these questions so that I can deliver them during one-on- one sessions to further assess employee knowledge. YOUSEF: I’ve been there where I had to just chance it with such assessments in a corporate environment. It’s admirable, what you do to ensure that employee knowledge is kept through such scenarios. Can you describe the nature of questions that you end up creating for this purpose? SHAYNA: They are mostly open-ended questions that are answered verbally, simulate customer facing situations and performing interaction role plays. YOUSEF: You mentioned how you need to export numbers and perform some tweaking to the data. How would you normally filter the data by? SHAYNA: I typically filter the data by course for all employees as this would be part of my own reporting. I also filter data per users for coaching sessions. YOUSEF: If I were to tell you that you have a magic wand and that you can change anything about what you currently use, what would you change? SHAYBA: I would want a feature to aggregate data from multiple courses together so that I can immediately gather data for each employee rather than having to stitch multiple reports on Excel for this matter. YOUSEF: I truly appreciate your time during this interview, you have been very generous with your answers. Is there anything else that you would like to ask me today? SHAYNA: It’s my pleasure. Not currently. Good luck with your research.

APPENDIX C: INTERVIEW TRANSCRIPT 3

Contact: Stella Bellou Role: Learning Consultant at Pentos AG Location: Munich, Germany Category: Use of LMS platforms within a corporate training environment Date: November 11th, 2018

YOUSEF: Hello Stella, I appreciate you taking the time for this interview today. I will provide a brief description about what we are working on so you can have further context about what we will be discussing today. We realize your background, especially with your experience in using SAP10 and SuccessFactors11, which is more focused for education within the corporate environment. I am currently a master’s student at the California State University, and this is part of my work towards my thesis. My work revolves around addressing Learning Management Systems and swivel seat scenarios observed by any stakeholder using an LMS platform alongside other applications, such as productivity apps or otherwise, where users are required to enter information in multiple locations, such as a word document and to a centralized repository, or exporting files from a system to save data into another file structure. I look forward to discussing similar pain points that you have observed. While I know about programming, I don’t know much about the user experience with LMS platforms, which is where we can use your expertise today. Can you please tell us about yourself and what you do? STELLA: First, thank you, I am happy to participate, it is interesting to me to get to know more about new LMS platforms. So far, I have been working the last year with the SuccessFactors learning platform. I am an implementation consultant, working with implementing, configuring the platform, rolling it out with different customers and countries. I also have a background in learning; I have studied psychology and learning. That’s an introduction about me, I hope this is enough. YOUSEF: This gives us a great baseline to start with. You have a learning background and you have interfaced first-hand with learning platforms. You mentioned that your role lies within implementation. Does that entail infrastructure and resources such as servers? Or do you mean training trainers and users? STELLA: Rather the second. What I personally do is configuring the instance such as creating roles and domains. I am rather on the business side rather than the technical side.

10 SAP is a multinational corporation based in Germany that specializes in enterprise software 11 SuccessFactors is a human resources management platform that includes a training system, developed by SAP SuccessFactors

I provide training to key users, collect requirements. I have colleagues with a deeper technical background who handle infrastructure, integrations, data imports, and so on. YOUSEF: I’m happy to hear that. We are looking to investigate more about the user experience side rather than mechanics at this point. I do understand that SuccessFactors is more geared towards a corporate environment, but what is the demographic of your customers? STELLA: I haven’t had any experience with educational organizations, but rather business organizations. While educational institutions do use SuccessFactors, but for employee training purposes rather than classroom delivery. YOUSEF: You mentioned something about training users, and creating roles, and domains, and tailor instances of SuccessFactors are tailored to your customer’s needs. Is it accurate to say you work as a vendor that distributes SuccessFactors? STELLA: Exactly. YOUSEF: When you sit down and meet with a customer, whether it is a trainer, admin, or otherwise; what are some questions or demands that you see users asking, in terms of ‘How can we do this using SuccessFactors’? STELLA: My answer would depend on what user you are referring to. Main stakeholders of such projects are rather project managers, or with HR roles within their organization. They are not the main users of the SuccessFactors instance. The real-life requirements and real questions come from key users who are typically representing a country, or business unit in their organization, and they have to serve their population. Typical challenges include the admin interface. This particular population of users may find the Admin interface, while improved comparing to other platforms, is not considered user friendly. YOUSEF: You mentioned stakeholders who are not using SuccessFactors on a daily basis, requesting strategic requirements for functionality and implementation, while key users may provide insight regarding executing tasks and handling the interface. Would this make sense? STELLA: In other words, key stakeholder are global managers. They typically do not have any knowledge on using the system, but rather requirements that fit their business needs and objectives and coordinate the project. They would not go to the system to create training content or register employees to content. These are activities that key users would do, who are also referred to as local admins; they are restricted to their local population. YOUSEF: Let’s discuss global administrators. They request overarching, business related inquiries. Can you provide an example of what they would ask for? STELLA: The number one requirement involves the end-user interface. Another common requirement is mobile access to the platform, especially to cover needs for employees working within plants as well as technicians. YOUSEF: You’re referring to first-line workers who may not have a desktop or computer environment. Instead they will need to use a shared device, tablet or a mobile device to access learning content.

STELLA: Yes. YOUSEF: If we are to shift our focus to local administrators, as more direct users of the platform, what would they typically ask for? STELLA: Easy reporting. An easy registration process. They typically ask for an intuitive experience using the system. This covers users with a role that is very particular purpose within the system; to deliver the training, collect signatures, participation, record learnings, initiate evaluations and so on. The challenges for this population of instructors is that we don’t train users or instructors, training is typically provided to admins. However, from previous business cases, we observed that challenges are faced while managing content. The training interface does not have access to edit content. Easy access to content repositories has been another big ask from users. SuccessFactors, as well as other LMS platforms, are typically connected to a content server, which is not considered a content repository. Trainers typically have folders with documents and manuals that they want to teach, and this is usually how they have their material. YOUSEF: So, you’re informing me that there is a bit of discord between where the content exists and how to load it into the LMS platform. STELLA: That is correct, this also pertains to educating users on how to use such features to import content. Many times, trainers will need to obtain some admin rights access to access content of training material to add PDFs, manuals and technical information that must be taught. Such organizations often keep such content within another storage system, such as SharePoint12, which is not easy to switch with the LMS platform. YOUSEF: To reiterate what you just mentioned, educators essentially find themselves in a tough spot to manage their learning content due to admin rights access, and being able to integrate their own learning content stored elsewhere with the LMS platform. STELLA: The integration is there, but it is not as how users would want to see it. YOUSEF: You also mentioned that educators tend to work with plenty of documents and PDF files stored elsewhere. Typically, what is the source of such content? STELLA: Content is often produced from within the same company for product training, such as training based on manuals. This is usually outsourced to an external vendor that creates course content using content authoring tools such as iSpring or Articulate13, as well as an SAP authoring tool. YOUSEF: What kind of course standards does SuccessFactors comply with? STELA: There are multiple standards, such as SCORM 1.2, SCORM 4.4, AICC, there are certain standards, with authoring tools that produce content within these specific formats. YOUSEF: What other applications would your users and their trainees need to leverage in conjunction with SuccessFactors to be able to get the job done? In other words, what else would be running on their computer next to SuccessFactors to achieve a certain goal?

12 Microsoft SharePoint is a shared storage service developed by Microsoft 13 iSpring and Articulate are eLearning content authoring applications

STELLA: Some storage repository, such as SharePoint, other platforms that show user data, Microsoft Word, PDF readers, content authoring tools, Camtasia14, Captivate15, and of course an email client. YOUSEF: Talking about email clients, is there any integration between SuccessFactors and email clients? STELLA: Partially; there are calendar events being sent out for registration and events which get tied to Outlook, but there is no way back to communicate from Outlook into SuccessFactors, for example, ignoring or declining an invitation does not reflect within SuccessFactors, which remains to be a pain point for some end users who are used to just accept or decline invitations for trainings due to limited access to SuccessFactors, however, this is to be addressed in the future. YOUSEF: You previously mentioned that easy reporting is a main ask from your users, can you provide an example of that? And is the issue related to how data is visualized, or what content the data carry? STELLA: There are lots of standard reports generated within the platform, such as user details, learning history and credits, certain analytics that need to be communicated to other parties. There are about 100 standards reports to use from, with certain filters and criteria to choose from, which can be exported using CSV or XTM formats with standard columns. In many cases, users may not be able to view certain fields, which lead them to create custom reports. As a partner, we provide the service to create tailor-made reports that fit customer needs. Customers usually have their own custom reports to use for such purposes. YOUSEF: Would you also find your users using exported CSV files in other programs such as Excel to further filter and process such data from a static data set? STELA: Yes, this is not uncommon. YOUSEF: To be focused into productivity applications, what is a typical use case scenarios for such programs, such as Word and Excel? STELLA: Excel is largely used for reporting. I can’t think of instances where Microsoft Word is used at this point. YOUSEF: Stella, you have been most helpful during this interview. I thank you for your time and helpfulness. Before we wrap up our interview, is there anything else you feel I should be asking you? STELLA: Nothing comes to my mind, I feel I shared most use cases that are commonly shared. I think we covered at least a high percentage of what I have observed users using in conjunction with SuccessFactors. YOUSEF: Sounds great. At this point, I am still working on a proof of concept that combines LMS platforms alongside productivity apps to address similar scenarios, and I would love to present you with a version to assess once available.

14 Camtasia is a screen video capturing program 15 Captivate is an eLearning authoring tool developed by Adobe

STELLA: That would be great, I would love to try that. YOUSEF: Thank you again for your time.