<p> Proposal for CIS8389: Directed Reading</p><p>Student: Linda Lee Professor: Alfred Kahn </p><p>Description of assignments/activities for course:</p><p>Title of Assignment: Measuring Systems Usage Without Method Bias </p><p>Purpose: To gain an in-depth understanding of the multi-method approach and to apply this understanding to studying systems usage. </p><p>Overview: The objective is to write a measurement paper on systems usage that would be acceptable for submission to a leading IS journal. </p><p>The emphasis will be to investigate the systems usage construct using a multi-method approach. In the past, systems usage has been addressed primarily using self-reported measures or, occasionally, objective data. Very rarely has systems usage been studied using a multi-method approach. </p><p>The student would need to complete reading in two areas: 1. Systems usage literature (empirical and theoretical) 2. Measurement literature </p><p>In Fall 2002, Linda and Alfred collected multi-method data from an experiment testing the relationship between systems usage and individual performance in financial analysis tasks. The data for this experiment is currently being coded. </p><p>The aim of this course would be for Linda to: - read the measurement literature in depth - develop a thorough understanding of the multimethod approach to measurement - apply this understanding to measuring systems usage and performance - determine how to analyze data obtained via multiple methods - conduct the analysis - write up a paper based on the analysis A high-level plan of activities and deliverables in the Summer session is: </p><p> Week of June 9: Measurement Theory 1 o Classic measurement theory readings (from list below) o Prepare annotated bibliography of classic readings o Summarize lessons from classical measurement theory for measuring systems usage o Annotated bibliography and summary of lessons learned: 5%</p><p> Week of June 16: Measurement Theory 2 o Modern measurement theory readings (from list below) o Prepare annotated bibliography of modern readings o Summarize lessons from modern measurement theory for measuring systems usage o Annotated bibliography and summary of lessons learned: 5%</p><p> Week of June 23: Sequence Analysis 1 o Sequence analysis readings (from list below) o Prepare annotated bibliography of sequence analysis readings o Summarize lessons from sequence analysis for measuring systems usage o Annotated bibliography and summary of lessons learned: 5%</p><p> Week of June 30: Sequence Analysis 2 o Continue sequence analysis readings (from list below) o Start practicing the sequence analysis program (URL listed below) o Continue annotated bibliography of sequence analysis readings o Write summary of expected approach to analyzing sequence data o Annotated bibliography and summary of analysis approach: 5%</p><p> Week of July 7: Multimethod 1 o Classic multimethod readings (from list below) o Prepare annotated bibliography of classic multimethod readings o Summarize lessons from multimethod readings for measuring systems usage o Annotated bibliography and summary of lessons learned: 5%</p><p> Week of July 14: Multimethod 2 o Modern multimethod readings (from list below) o Prepare annotated bibliography of modern readings o Summarize lessons from modern multimethod literature for systems usage o Annotated bibliography and summary of lessons learned: 5%</p><p> Week of July 21: Manuscript Preparation 1 o Finalize data analysis of multimethod data o Present rough draft of final paper including the analysis of the self- reported data and the objective data o Draft of paper: 20%</p><p> Week of July 28: Manuscript Preparation 2 o Present final version of paper including the analysis of the self-reported data and the objective data o Final paper: 40%</p><p>Participation in Meetings: 10%</p><p>Grades due: August 6. </p><p>Readings</p><p>1. Measurement theory readings: </p><p>Books: Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Hott, Rinehart, and Winston.</p><p>Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. New York: McGraw- Hill</p><p>Articles: Cronbach, L.J. & Meehl, P. Construct validity in psychological tests. Psychological Bulletin, 52, 1955, 281-302.</p><p>Englehard, G. (1984). Thorndike, Thurstone, and Rasch: A Comparison of their methods of scaling psychological and educational tests. Applied Psychological Measurement, 21- 38.</p><p>Jaeger, R.M. (1987). Two decades of revolution in educational measurement, Educational Measurement: Issues and Practice, 6-14.</p><p>Osburn, H. G. (2000). Coefficient alpha and related internal consistency reliability coefficients. Psychological Methods, 5, 343-355. Michell, J. (1986). Measurement scales and statistics: A clash of paradigms. Psychological Bulletin, 100, 398-407.</p><p>Michell, J. (1997). Quantitative science and the definition of measurement in psychology. British Journal of Psychology, 88, 355-383.</p><p>Miller, M. B. (1995). Coefficient alpha: A basic introduction from the perspectives of classical test theory and structural equation modeling. Structural Equation Modeling, 2, 255-273.</p><p>Mislevy, R.L. (1996). Test theory reconceived. Journal of Educational Measurement, 379-416.</p><p>Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199-223.</p><p>Sussman, M., & Robertson, D. U. (1986). The validity of validity: An analysis of validation study designs. Journal of Applied Psychology, 71, 461-468.</p><p>Traub, R.E. & Rowley, G.L. (1991). Understanding reliability. Educational Measurement: Issues and Practice, 10, 37-45.</p><p>Traub. R.E. (1997). Classical test theory in historical perspective. Educational Measurement: Issues and Practice, 8-14.</p><p>Wainer, H. (1989). The future of item analysis. Journal of Educational Measurement,26, 191-208.</p><p>2a. Sequence Analysis Readings: </p><p>The approach to analyzing sequence data proposed by Linda Abbott will be studied. A selection of his papers are below. Elena Karahanna suggested that we use his approach for analyzing objective IS usage data.</p><p>"Event Sequence and Event Duration: Colligation and Measurement" Historical Methods, 17:192-204, 1984. </p><p>"Optimal Matching Methods for Historical Data" with John Forrest Journal of Interdisciplinary History, 16:473-496, 1986.</p><p>"A Primer on Sequence Methods." Organization Science, 1:373-392, 1990. </p><p>"Measuring Resemblance in Social Sequences" with Alexandra Hrycak American Journal of Sociology, 96:144-185, 1990. "The Optimal Matching Method for Anthropological Data: An Introduction and Reliability Analysis" with John Forrest Journal of Quantitative Anthropology 2:151-170, 1990. </p><p>"From Causes to Events" Sociological Methods and Research, 20:428-455, 1992. </p><p>"Sequence Analysis" Annual Review of Sociology, 21:93-113, 1995. </p><p>"Sequence Analysis and Optimal Matching Methods in Sociology" with Angela Tsay Sociological Methods and Research, 29:3-33, 2000</p><p>"Reply to Levine and Wu" Sociological Methods and Research, 29:65-76, 2000</p><p>2b. Computer program for Sequence Analysis: (download site): http://www.src.uchicago.edu/users/abbot//om.html</p><p>3. MultiTrait MultiMethod Readings: </p><p>Bagozzi, R. P., & Yi, Y. (1990). Assessing method variance in multitrait-multimethod matrices: The case of self-reported affect and perceptions at work. Journal of Applied Psychology, 75, 547-560.</p><p>Bagozzi, R. P., & Yi, Y. (1991). Multitrait-multimethod matrices in consumer research. Journal of Consumer Research, 17, 426-439.</p><p>Bagozzi, R. P., Yi, Y., & Phillips, L. W. (1991) Assessing construct validity in organizational research. Administrative Science Quarterly, 36, 421-458.</p><p>Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.</p><p>Cudeck, R. (1988). Multiplicative models and MTMM matrices. Journal of Educational Statistics, 13, 131-147.</p><p>Fiske, D. W., & Campbell, D. T. (1992). Citations do not solve problems. Psychological Bulletin, 112, 393-395.</p><p>Glick, W. H., Jenkins, D. G., & Gupta., N. (1986). Method versus substance: How strong are underlying relationships between job characteristics and attitudinal outcomes. Academy of Management Journal, 29, 441-484. Golding, S. L., & Seidman, E. (1974). Analysis of multitrait-multimethod matrices: A two step principal components procedure. Multivariate Behavioral Research, 9, 479-496. </p><p>Graham, J. W., & Collins, N. L. (1991). Controlling correlational bias via confirmatory factor analysis of MTMM data. Multivariate Behavioral Research, 26, 607-629.</p><p>Hammond, K. R., Hamm, R. M., & Grassia, J. Generalizing over conditions by combining the multitrait-multimethod matrix and the representative design of experiments. Psychological Bulletin, 1986, 100, 257-269.</p><p>Lee, R., Malone, M., & Greco, S. Multitrait-Multimethod-Multirater Analysis of Performance Ratings for Law Enforcement Personnel. Journal of Applied Psychology, 1981, 66, 625-632.</p><p>Marsh, H. W. (1993a). Multitrait-multimethod analysis: Inferring each trait-method combination with multiple indicators. Applied Measurement in Education, 6 (1), 49-81.</p><p>Marsh, H. W., & Bailey, M. (1991). Confirmatory factor analysis of multitrait- multimethod data: A comparison of alternative models. Applied Psychological Measurement, 15, 47-70.</p><p>Millsap, R. E. (1990). A cautionary note on the detection of method variance in multitrait-multimethod data. Journal of Applied Psychology, 75, 350-353. </p>