Preliminary Draft for Discussion

Integrating Technology into Evaluation to Produce Truth: Past, Present & Future

Paul Lorton, Jr. Moira A Gunn University of San Francisco

Abstract The truth we can speak to power is a function of the veracity of the information we present. The interest of those whose concern is the technology employed in producing that information must be in how that technology can add to the veracity value. In this discussion technology employed in the past, available at present and that can be forecast for the future will be discussed in how it has, can and will contribute to creating evaluative information which speaks truth. In exploring this topic we will follow our introductory prologue about the nature of truth with explorations of technology in the past (e.g. telephones, mark sense data gathering), the present (e.g. Survey Monkey, cell phones) and the uncertain, rapidly changing future. We end with a brief epilogue on the ethical, societal and legal constraints on our efforts.

Introduction

The concern of those interested in employing technology in evaluation ought to be that the technology be employed appropriately and effectively. Technology has always be a tool and conceived as a servant to our efforts to control in one way or another the world around us. In the context of the theme of this conference, “Speaking Truth to Power” that control should be employed in revealing “truth.”

“What is truth?” is certainly a question we must deal with before we can seek to articulate it to authority. In this discussion, as a written prologue, we will start with shining a light on the path to answering that question with the full ambiguity contained in most academic answers (see, for example, http://www.philosophynews.com/post/2015/01/29/What-is-Truth.aspx)

To frame briefly the issue of this discussion, consider the dictionary note: Truth (noun) 1.the quality of being true, genuine, actual, or factual: the truth of his statement was attested 2.something that is true as opposed to false: you did not tell me the truth 3.a proven or verified principle or statement; fact: the truths of astronomy1

For those concerned with evaluation, the third definition is the most relevant for in evaluation truth is an epistemological issue to be addressed empirically. Thus the way truth is attainted for evaluation is by the reliable and valid assessment of the external world evaluated.

Can that be done? For those whose interest is the employment of technology in that task, the belief is that technology can be employed in evaluation to improve reliability thence validity to yield truth. This was certainly the belief in the infamous Liberty Digest poll for the 1936 presidential election:

1 (from: truth. (n.d.). Collins English Dictionary - Complete & Unabridged 10th Edition. Retrieved March 14, 2018 from Dictionary.com website http://www.dictionary.com/browse/truth) AEA 2018 ITE TIG Business Meeting

The mailing list for the publication’s 1936 straw vote poll was culled from automobile registrations and telephones books, resulting in about 10 million names. Approximately 2.4 million recipients returned the mock ballots. (see http://www.proquest.com/blog/pqblog/2016/That-Time- the-Literary-Digest-Poll-Got-1936-Election-Wrong.html)

The largest, most expensive poll ever undertaken to that point used then current technology – automobile ownership, telephone books - to produce what would have been very believable if the results had been “true.”

In examining technology employed in evaluation we must consider how that technology helps yield veracity. For this discussion, technology used in evaluation from the past, much from presentations over the past 35 AEA conferences, will be considered with this issue in mind. What did mark-sense response gathering contribute to reliability? Validity? For technologies used at present, the focus will shift to the technologies employed during the most recent half decade, again drawing on what has been presented in AEA sessions for this TIG. Again, how would applications such as Survey Monkey contribute to reliability? Validity? Finally, the future of technology in evaluation will be considered by looking at the rapidly evolving precision and intrusiveness of technology and how it can improve the reliability, validity and veracity of what evaluation can communicate to authority.

Technology’s Role

The role of technology in evaluation and, notably, the AEA interest in technology can be defined by the focus from the last half of the 20th century until the present on the transistorization of our world. The perfection of the long sought solid-state replacement for bulky, costly, and short- lived vacuum tubes (or valves) was demonstrated December 23, 1947 and the team that demonstrated it was awarded the Nobel prize in Physics in 1956. In 1965, Gordon Moore published a paper2 observing that the complexity of integrated transistor circuits was increasing by a factor of two every year. In that article, Moore predicted, “That means by 1975, the number of components per integrated circuit for minimum cost will be 65,000. “ Now, a little over 40 years from Moore’s statement, the processor chip in the latest iPhone (Apple’s A12) has 6.9 billion transistors. Other chips have even greater densities but the Apple SOC is in a phone forcing our attention on the range of applications possible in ordinary tasks.

By harnessing and directing the increasing capabilities of electronic circuits on tasks which are necessary to evaluation – data collection, analysis, interpretation – the role technology plays can increase as the capability of the circuits increases.

Past

The past of our effort in employing technology in evaluation can be summarized most appropriately in the that the passage of time compelled the change in name of this TIG from Computer Uses in Evaluation to its current title Integrating Technology into Evaluation for the 2004 Conference in Atlanta.

2 Moore, Gordon E. “Cramming more components onto integrated circuits.” Electronics, 38, #8, April 19, 1965

Preliminary Draft for Discussion 2 AEA 2018 ITE TIG Business Meeting

The early efforts in integrating technology in evaluation followed much the same path that marked Babbage’s 19th century on the Difference Engine. The purpose of that project was to replace human “computers” with a mechanical mean for producing mathematical tables (trigonometric values, logarithms, etc.) as human produced tables were rife with errors. From the first successful technology to record values in machine readable form (Hollerith and the tabulating machines for the US 1890 census), the strong realization that getting under mechanical control as close to where they were created was going to improve the accuracy of subsequent information drawn from those numbers3.

While the Apple had easy to use and a graphical user interface from 1984, it was 1995 with the release in August of that year of Windows 95 and the integrated “WinTel” version of Microsoft Office that got the use of computers in evaluation more exposure as a tool. For the 1999 Conference in Orlando, several session sponsored by the Computer Use TIG focused on on-line resources for evaluators as well as on-line surveys. The trend of using microcomputers had been growing slowly replacing the earlier efforts employing terminals from large system directly to administer instruments or as repositories of data collected via pencil and paper.

The following year, in Honolulu with “Evaluation Capacity Building” as the presidential theme, several sessions sponsored by the TIG focused on that, including sessions titled:

Building Evaluation Capacity Byte by Byte: More Computer-based and On-line Resources for Evaluators and Casting the Net on New Possibilities in Data Collection and Reporting

Another multipaper session highlights the growing sophistication of software solutions for analyzing the responses to surveys and of using sophisticated software for analysis. A session entitled “Software Solutions to Applied Research Problems “ included the following presentations:

Electronic Surveys with Survey Solutions Analyzing Spatial Data with ArcView GIS, Entering Data with Remark Importing SAS Data Files into SPSS

In 2003, the last year the TIG was defined as “Computer Uses”, the programs included presentations titled:

Session Title: Geographic Information Systems, Geography and Gender Equity in Social Programs • Geographic Information Systems (GIS) Methods Using Neighborhood Attributes and Survey Data for Program Evaluation

3 Note: A converting reality into numbers

The Babylonians had a numbering system based on 60 growing from the use of the 12 knuckles of one human hand and the five fingers of the other as a starting point. In order to help reality conform to this numbering system, they constructed a calendar of 360 days (6*60) and divided a circle into 360 degrees from whence also came 60 minutes in an hour and so on. Much of that system was subsequently abandoned. Currently the numbering system that governs our digitization of every piece of reality from music to visual imagers to self- driving automobiles is much simpler: 0 & 1. That system and the Boolean algebra once thought (by George Boole) to model human thought are the basis of the immense computational power we can bring to bear on every human task.

Preliminary Draft for Discussion 3 AEA 2018 ITE TIG Business Meeting

• Geographic Information Systems (GIS) Methodology in an Evaluation of Community-based Programs for High-risk Female Youth

Session Title: Building an Evaluation Infrastructure Within Federally Funded Teacher Education Programs: a Case Study of Systemic Contexts in Designing, Selecting and Using Evaluation Methods

Session Title: Building Evaluation Capacity in our Schools and the Larger Community Through Advanced Technologies (GIS, Handhelds and the Internet) • The Hand-held Computer as a Data Collection Tool in Evaluation • Internet Data Collection: The Final Frontier, A Portrait of a Web-based Holocaust Education Assessment

Session Title: Using Technology to Build Bridges-Using Technology to Build Theory • Issues of Data Collection in Using Technology and Collaborative Approaches to Bridge the Resource Gap • Using Software to Improve Program Theory and Causal Reasoning

Session Title: Simple and Affordable Online Data Collection You Can Do

Session Title: Evaluating Website Performance: A Formative Evaluation of the Online Evaluation Resource Library (OERL)4 • Expert Panel Review of the Online Evaluation Resource Library's (OERL's) Instructional Utility • Usability Testing of Online Evaluation Resource Library (OERL) • Surveying Online Evaluation Resource Library (OERL) Users

This collection of presentations illustrate that the state of the TIGs interest has grown from a focus on using computers per se to more of a focus on the software available yet other forms of technology are starting to appear (e.g. “hand-held devices”)

The following year 2004 in Atlanta, when the focus of the TIG broadened to be Technology and not simply computer uses, the session included:

• Save a Tree: Using Technology to Collect, Manage, Analyze, and Evaluate Your Data • Online Data Collection: Balancing Security With Explicit Information Needs • Online Tools: Groundbreaking Strategies for Capturing and Analyzing Student Performance • Capturing the Voices: The Advantages and Challenges of Online Feedback • A Picture's Worth a Thousand Words: Using New Technologies to Enhance Evaluation with Pictures, Photos and Video • Did We Make a Difference? Community Indicators, Evaluation and Technology: Tools to Evaluate Change at the Neighborhood Level

By 2006 in Portland, the session more completely reflect issues dealing with technology in supporting evaluation than in the early years of the TIG:

• Matrix Assessment: Tools and Methods for Bridging Local Practice to Institutional Reporting Need • Integrating Technology Into Evaluation to Study and Support Executive Development Training Impact

4 OERL still has a web presence (https://oerl.sri.com/home.html) although the most recent articles are from 2007

Preliminary Draft for Discussion 4 AEA 2018 ITE TIG Business Meeting

• Using Technology to Showcase School Performance: How Artful Design and Research Insight Improve Data Collection and Reporting • Online Surveys: Thinking Through Design Nuances to Ensure Accurate Results • Integrating Technology Into Evaluation TIG Business Meeting and Presentation: The Evaluative Challenges of Social Networking: The 'MySpace' Phenomenon • Using Web-based Technology to Evaluate and Support Community Collaborations • Using Technology to Creatively and Robustly Manage and Monitor Complex Projects • Advanced Online Survey Development and Use • Putting the 'Byte' in Data Collection and Reporting: Strategies for Using Technology to Collect, Analyze and Report Data • Formative Research in Virtual Communities: Principles, Methods and Tools • Innovative Solutions in Monitoring and Tracking Using Technology-Based Data Management Systems

Up to the present, say the last 5 or 10 years, the effort in Using Computers or Integrating Technology into evaluation has been to achieve more closely the dream of Babbage in the Difference Engine, to reduce the human error factor in handling numbers. Many of the reports shared at the AEA conferences focused on this – mechanical data collection and the machine handling of those data once they took electronic form. In our search for “truth,” the influence of technology was to increase the sense that the data were accurate and their handling was reliable.

Present

The present is a mixed bag of older techniques in emerging contexts and innovations with much promise. The direction of technology in recent years has seen the remarkable 10 years of the iPhone and smarter and smarter hand-held devices so overwhelm our world that hand-held screen time has taken over our former concern over couch-potato time. Phones and tablets are the primary access point for data collection. A glance at some recent session titles will illustrate this:

AEA 2016 ITE Sessions

Session Number: ITE1 Abstract 1 Title: Contribution of the Internet of Things to Evaluation Design: Prospects and Problems Abstract 2 Title: Designing Protocols to Maximize Real-Time Data Collection and Data Use Abstract 3 Title: Two Birds, One Technical Assistance (TA) Form: Designing a mobile TA monitoring system with dual functionality for both Provider and Evaluator

Session Number: ITE2 Abstract 1 Title: Establishing an organization-wide monitoring and evaluation system: It’s not really about the technology Abstract 2 Title: One Year Later: Applying Lessons Learned in the Ongoing Design of an M&E Technology Platform for International Development Projects

“You Can’t Do It with a Spreadsheet: A Discussion on Database Creation”

Exploring the innovations, the challenges and the possibilities of using Cloud Based Management Information Systems (CMIS) to plan and evaluate development interventions

Fundamentals of Database Management: Introduction to Microsoft Access

Utilizing Online Technology Tools (UserTesting.com, Google Analytics, ArcGIS Online and Smartsheet) to Support Evaluation

Preliminary Draft for Discussion 5 AEA 2018 ITE TIG Business Meeting

“It’s all about the Tweets”: Designing and Implementing a Social Media Evaluation

The present sessions sponsored by the ITE TIG have focused on refining the capabilities of already proven worth with topics and workshops which share the contribution of computer applications such as database systems, spreadsheets and analysis software.

Within this collection are also shadows from the future with discussions of “the cloud,” social media, Google analytics and other approaches quite new but with considerable promise.

Future

The future will bring many innovative and powerful ways in which technology can be employed in evaluation. Four technologies can be marked now for their future impact: Social Media, the Cloud, “big data/analytics” and the Internet of Things.

Social Media

In conjunction with the cloud, which is where social media lives, and big data/analytics, which is where it achieves its impact, Social Media (Instagram, LinkedIn, Facebook, Twitter, Snapchat, etc.) can be a very productive tool but, because it is social and publicly exposed, it is also very fragile. It can be a valuable tool through which important data can be gleaned but the intrusion of external forces on it from both legitimate public policy concerns and illegitimate “hacking” it must be monitored closely. Portals such as these can come and go over night. (Did you back up your MySpace data?)

The Cloud

The naming of the cloud is murky and has many authors. The most cited is a Compaq computer internal presentation5 from November 1996 but the idea of computing and storage in some amorphous (but physically certain) electronic space dates back to at least the first instance of remote computing. That it been made real by the internet, the world wide web, inexpensive and remotely accessible mass storage and the power of electronic data manipulators to do wonders over broadband communication lines is where we are today.

For evaluation efforts, the use of the cloud is entrenched and, in most cases, invisible. Almost everything we do on-line is “clouded” and we may not explicitly realize it or be prepared for the consequences. However, clouding data makes it more accessible for our constructive purposes and for use in the future feature.

Big Data/Analytics

To quote from the IBM Analytics site:

Analyzing big data allows analysts, researchers, and business users to make better and faster decisions using data that was previously inaccessible or unusable. Using advanced analytics techniques such as text analytics, machine learning, predictive analytics, data mining, statistics, and natural language processing, businesses can analyze previously untapped data sources independent

5 see https://s3.amazonaws.com/files.technologyreview.com/p/pub/legacy/compaq_cst_1996_0.pdf

Preliminary Draft for Discussion 6 AEA 2018 ITE TIG Business Meeting

or together with their existing enterprise data to gain new insights resulting in better and faster decisions.6

This quote, as is the case with most discussions of big data and the sophisticated analytics that are used to ferret out the hidden nuisances lying beyond our human vision within the vast quantity of numbers collected sounds very attractive and as a great gift to our further understanding of the effect of our interventions on the stakeholders and other targets of our efforts. The drawback is that both of these very great potential sources of further insight into the nature of us are beyond our control and possibly our understanding. What the personalization of computing from the microcomputer to the smart phone has done to place data processing power in our hands big data and the analytic tools to master that source of information has placed back in the hands of those controlling the machine room as it was in the early decades of computing.

The Internet of Things

Let us look at this description:

The W1 chip is designed in-house by Apple based on existing Bluetooth technology. Matter of fact, it uses Bluetooth 4.1 for connectivity. However, the W1 is much more than just a custom Bluetooth chipset. Apple has added tech to it that makes pairing effortless when you pair it with supported devices. Normally the pairing process between any Bluetooth headset and a device like a smartphone is a bit cumbersome, requiring you to open up your Bluetooth settings and manually select the headphones you want to pair. But thanks to tech in the W1 chip, that manual-pairing requirement is history. As soon as the W1 detects your iPhone—even for the first time—it pairs with it within second (without you having to do a thing.7

In more technical detail, another description lists the hardware and then the uses at present.

• sensors can calculate when one (or both) of your AirPods buds are in your ear • Infra-red sensors are the small dots you can see on either side of the AirPod’s speaker ports. They work together with the motion sensors to work out when you've put a bud in your ear • Voice accelerometers recognise when you’re speaking, so when you're on the phone, or speaking to , your music won't get in the way

With the W1 chip managing the sensors mentioned above, AirPods are capable of doing some groundbreaking stuff: • They can work out whether they’re in or out of your ear. Because of this, they can pause playback when you take one out. • They can detect when you’re talking, and turn on microphones to hear what you’re saying. • When you’re speaking, they can eliminate background noises to hear what you’re saying better. • They can let you access Siri with a simple double-tap. You can then ask Siri to control your AirPods’ volume. • They can sync across lots of different devices. Just open your AirPods’ carrying case (which doubles as a charging case) and they’ll be paired with all the devices that linked to your iCloud account.8

6 see https://www.ibm.com/analytics/hadoop/big-data-analytics 7 http://www.knowyourmobile.com/earphones/24411/apples-w1-airpod-chip-explained-apple-airpod-deals 8 https://lowdown.carphonewarehouse.com/feature/what-does-the-w1-chip-do/37611/

Preliminary Draft for Discussion 7 AEA 2018 ITE TIG Business Meeting

These quotes are about Apple’s AirPods. From this and the descriptions of many other devices in the Internet of Things, the marvel is what can be done with digital circuitry now. The processor and the integrated sensors are inert withy out the software, the code, to guide them through their tricks. That code grows as the coders gain experience with devising tasks for these devices.

Expectation for Integrated Technology

Before we close this discussion with some human considerations beyond simply employing technology to squeeze meaning out of numbers in the epilogue, a word about what should be expected of our ability to bring increasingly sophisticated technology to bear on observing and recording human activity.

The degree to which digital devices will be able to monitor and record our activities will increase beyond our imagination and be limited only by that imagination to present it with tasks. The hardest task in developing things for the internet of things to encompass is thinking of that “thing.” A little over 10 years ago Apple unveiled the iPhone. At the time the major players in the cell phone sphere were, first of all, Motorola and largely Nokia. Now both of those vital companies are a footnote (all be it significant) in the history of mobile communications. The next ten years will probably bring similar changes in technology and its every day use. This cannot but have a profound and positive effect on our use of technology to produce increasingly recognized “truth.”

Epilogue

The most positive view of the future of integrating technology into evaluation is that the technology will provide the leverage and facility to deliver accurate, reliable data collection and reduction and deliver the resulting information within the limits of mechanical validity. The human task in this will be to make sure the non-mechanical assurance of validity; the essence of the truth we wish to deliver is there while also addressing the social and political constraints.

In their conclusion to a 2016 AEA paper on issues to be expected in using the Internet of Things in Evaluation, we concluded:

The biggest challenge is the ethical concern one must have in where and when to use the increasingly invisible power of current and future technology in evaluation. We did put the cart before this horse and responded to this challenge in the AEA conference in Chicago, 2015. Gunn & Lorton9 addressed the ethical issues raised by the sophisticated and connected devices available for tracking, advances in which are the issue in this consideration also. Many companies, organizations and agencies (Apple, AEA, US Congress via HIPPA) have positive positions on how to anticipate the ethical issues and deal with them ahead of time (security also as ethics and security always seem to be issues demanding resolution after the product launch)10.

Again, to quote from the Lorton & Gunn 2016 discussion:

9 Gunn, Moira A & Paul Lorton, Jr. Ethics & Tracking via Technology – Getting Ahead of the Game. AEA Annual Conference, Chicago, Il. Nov 2015 10 Lorton, P. Jr. & Moira A Gunn Contribution of the Internet of Things to Evaluation Design: Prospects and Problems. AEA Annual Conference, Atlanta, GA. Oct 2016

Preliminary Draft for Discussion 8 AEA 2018 ITE TIG Business Meeting

Turner11 illustrates the double edged sword with connected wearables which have proven to be valuable in productivity increases but have raised issues about how much employers are able to invade the privacy of employees even though the data collected are quite valuable to all.

To repeat the quote from Tracy Staedter’s comment on emotion recognition software, “There’s a high level of creepiness here that cannot be denied…12”

We must trust our good sense to provide the care of the potential in using the burgeoning Internet of Things to enhance our evaluation efforts and avoid using the powerful monitoring tools for purposes unrestrained by our good judgment.

Finally, as a note for the continued success and contribution of the ITE TIG, technology is a tool with broad scope for applications. Using something new always has a learning curve and the topical interest group concerned with integrating these tools into evaluation is the venue where such learning can take place especially in the early stages of any evaluation effort.

11 Turner, Karen “Are performance-monitoring wearables an affront to workers’ rights?” - The Washington Post August 5, 2016 12 Staedter , Tracy “Face Scanning Tech Reveals Emotional Impact of Ads” News.Discovery.Com, Feb 12, 2016 (see http://News.Discovery.Com/Tracy-Staedter.Htm)

Preliminary Draft for Discussion 9