Proposal for CIS8389: Directed Reading

Student: Linda Lee Professor: Alfred Kahn

Description of assignments/activities for course:

Title of Assignment: Measuring Systems Usage Without Method Bias

Purpose: To gain an in-depth understanding of the multi-method approach and to apply this understanding to studying systems usage.

Overview: The objective is to write a measurement paper on systems usage that would be acceptable for submission to a leading IS journal.

The emphasis will be to investigate the systems usage construct using a multi-method approach. In the past, systems usage has been addressed primarily using self-reported measures or, occasionally, objective data. Very rarely has systems usage been studied using a multi-method approach.

The student would need to complete reading in two areas: 1. Systems usage literature (empirical and theoretical) 2. Measurement literature

In Fall 2002, Linda and Alfred collected multi-method data from an experiment testing the relationship between systems usage and individual performance in financial analysis tasks. The data for this experiment is currently being coded.

The aim of this course would be for Linda to: - read the measurement literature in depth - develop a thorough understanding of the multimethod approach to measurement - apply this understanding to measuring systems usage and performance - determine how to analyze data obtained via multiple methods - conduct the analysis - write up a paper based on the analysis A high-level plan of activities and deliverables in the Summer session is:

 Week of June 9: Measurement Theory 1 o Classic measurement theory readings (from list below) o Prepare annotated bibliography of classic readings o Summarize lessons from classical measurement theory for measuring systems usage o Annotated bibliography and summary of lessons learned: 5%

 Week of June 16: Measurement Theory 2 o Modern measurement theory readings (from list below) o Prepare annotated bibliography of modern readings o Summarize lessons from modern measurement theory for measuring systems usage o Annotated bibliography and summary of lessons learned: 5%

 Week of June 23: Sequence Analysis 1 o Sequence analysis readings (from list below) o Prepare annotated bibliography of sequence analysis readings o Summarize lessons from sequence analysis for measuring systems usage o Annotated bibliography and summary of lessons learned: 5%

 Week of June 30: Sequence Analysis 2 o Continue sequence analysis readings (from list below) o Start practicing the sequence analysis program (URL listed below) o Continue annotated bibliography of sequence analysis readings o Write summary of expected approach to analyzing sequence data o Annotated bibliography and summary of analysis approach: 5%

 Week of July 7: Multimethod 1 o Classic multimethod readings (from list below) o Prepare annotated bibliography of classic multimethod readings o Summarize lessons from multimethod readings for measuring systems usage o Annotated bibliography and summary of lessons learned: 5%

 Week of July 14: Multimethod 2 o Modern multimethod readings (from list below) o Prepare annotated bibliography of modern readings o Summarize lessons from modern multimethod literature for systems usage o Annotated bibliography and summary of lessons learned: 5%

 Week of July 21: Manuscript Preparation 1 o Finalize data analysis of multimethod data o Present rough draft of final paper including the analysis of the self- reported data and the objective data o Draft of paper: 20%

 Week of July 28: Manuscript Preparation 2 o Present final version of paper including the analysis of the self-reported data and the objective data o Final paper: 40%

Participation in Meetings: 10%

Grades due: August 6.

Readings

1. Measurement theory readings:

Books: Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Hott, Rinehart, and Winston.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. New York: McGraw- Hill

Articles: Cronbach, L.J. & Meehl, P. Construct validity in psychological tests. Psychological Bulletin, 52, 1955, 281-302.

Englehard, G. (1984). Thorndike, Thurstone, and Rasch: A Comparison of their methods of scaling psychological and educational tests. Applied Psychological Measurement, 21- 38.

Jaeger, R.M. (1987). Two decades of revolution in educational measurement, Educational Measurement: Issues and Practice, 6-14.

Osburn, H. G. (2000). Coefficient alpha and related internal consistency reliability coefficients. Psychological Methods, 5, 343-355. Michell, J. (1986). Measurement scales and statistics: A clash of paradigms. Psychological Bulletin, 100, 398-407.

Michell, J. (1997). Quantitative science and the definition of measurement in psychology. British Journal of Psychology, 88, 355-383.

Miller, M. B. (1995). Coefficient alpha: A basic introduction from the perspectives of classical test theory and structural equation modeling. Structural Equation Modeling, 2, 255-273.

Mislevy, R.L. (1996). Test theory reconceived. Journal of Educational Measurement, 379-416.

Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199-223.

Sussman, M., & Robertson, D. U. (1986). The validity of validity: An analysis of validation study designs. Journal of Applied Psychology, 71, 461-468.

Traub, R.E. & Rowley, G.L. (1991). Understanding reliability. Educational Measurement: Issues and Practice, 10, 37-45.

Traub. R.E. (1997). Classical test theory in historical perspective. Educational Measurement: Issues and Practice, 8-14.

Wainer, H. (1989). The future of item analysis. Journal of Educational Measurement,26, 191-208.

2a. Sequence Analysis Readings:

The approach to analyzing sequence data proposed by Linda Abbott will be studied. A selection of his papers are below. Elena Karahanna suggested that we use his approach for analyzing objective IS usage data.

"Event Sequence and Event Duration: Colligation and Measurement" Historical Methods, 17:192-204, 1984.

"Optimal Matching Methods for Historical Data" with John Forrest Journal of Interdisciplinary History, 16:473-496, 1986.

"A Primer on Sequence Methods." Organization Science, 1:373-392, 1990.

"Measuring Resemblance in Social Sequences" with Alexandra Hrycak American Journal of Sociology, 96:144-185, 1990. "The Optimal Matching Method for Anthropological Data: An Introduction and Reliability Analysis" with John Forrest Journal of Quantitative Anthropology 2:151-170, 1990.

"From Causes to Events" Sociological Methods and Research, 20:428-455, 1992.

"Sequence Analysis" Annual Review of Sociology, 21:93-113, 1995.

"Sequence Analysis and Optimal Matching Methods in Sociology" with Angela Tsay Sociological Methods and Research, 29:3-33, 2000

"Reply to Levine and Wu" Sociological Methods and Research, 29:65-76, 2000

2b. Computer program for Sequence Analysis: (download site): http://www.src.uchicago.edu/users/abbot//om.html

3. MultiTrait MultiMethod Readings:

Bagozzi, R. P., & Yi, Y. (1990). Assessing method variance in multitrait-multimethod matrices: The case of self-reported affect and perceptions at work. Journal of Applied Psychology, 75, 547-560.

Bagozzi, R. P., & Yi, Y. (1991). Multitrait-multimethod matrices in consumer research. Journal of Consumer Research, 17, 426-439.

Bagozzi, R. P., Yi, Y., & Phillips, L. W. (1991) Assessing construct validity in organizational research. Administrative Science Quarterly, 36, 421-458.

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.

Cudeck, R. (1988). Multiplicative models and MTMM matrices. Journal of Educational Statistics, 13, 131-147.

Fiske, D. W., & Campbell, D. T. (1992). Citations do not solve problems. Psychological Bulletin, 112, 393-395.

Glick, W. H., Jenkins, D. G., & Gupta., N. (1986). Method versus substance: How strong are underlying relationships between job characteristics and attitudinal outcomes. Academy of Management Journal, 29, 441-484. Golding, S. L., & Seidman, E. (1974). Analysis of multitrait-multimethod matrices: A two step principal components procedure. Multivariate Behavioral Research, 9, 479-496.

Graham, J. W., & Collins, N. L. (1991). Controlling correlational bias via confirmatory factor analysis of MTMM data. Multivariate Behavioral Research, 26, 607-629.

Hammond, K. R., Hamm, R. M., & Grassia, J. Generalizing over conditions by combining the multitrait-multimethod matrix and the representative design of experiments. Psychological Bulletin, 1986, 100, 257-269.

Lee, R., Malone, M., & Greco, S. Multitrait-Multimethod-Multirater Analysis of Performance Ratings for Law Enforcement Personnel. Journal of Applied Psychology, 1981, 66, 625-632.

Marsh, H. W. (1993a). Multitrait-multimethod analysis: Inferring each trait-method combination with multiple indicators. Applied Measurement in Education, 6 (1), 49-81.

Marsh, H. W., & Bailey, M. (1991). Confirmatory factor analysis of multitrait- multimethod data: A comparison of alternative models. Applied Psychological Measurement, 15, 47-70.

Millsap, R. E. (1990). A cautionary note on the detection of method variance in multitrait-multimethod data. Journal of Applied Psychology, 75, 350-353.