Measuring and Monitoring Movements in Course Interest at the University Computing Centre
Total Page:16
File Type:pdf, Size:1020Kb
Journal of Computing and Information Technology - CIT 9, 2001, 3, 197–205 197 Measuring and Monitoring Movements in Course Interest at the University Computing Centre: A Statistical Platform for Quality Improvement and New Course Deployment Decisions Vesna Luzar-Stifˇ fler1 2, Miroslav Milinovic´1 and Charles Stiffler2 1University Computing Centre, Zagreb, Croatia 2CAIR Research Centre, Zagreb, Croatia A sample of 7000 students was analyzed by the sity Computing Centre, Zagreb SRCE . The University Computing Centre SRCE staff to evaluate “non-credit” courses being offered to students, course satisfaction, teacher performance, and changing course demands. Findings indicated adequate overall faculty and staff at SRCE, have been devel- performance, and changing course content needs over oped over the years by both SRCE and CARNet a four year time period. The major conclusion of Croatian Academic and Research Network . the Project SOCRATES is that the software product developed during the research can become an econom- ical and useful tool in helping to shape administrator A. Background and Motivation: limited teach- efforts and decisions regarding the allocation of scarce ing resources coupled with increasing de- organizational resources and in the development of new mand for computer, software, and internet course offers at SRCE. literacy training at SRCE demanded the cre- ation of a more economical approach to the Keywords: statistical analysis, ANOM, courseteacher evaluation, SAS application, CRM forecasting and deployment of both current and new course offerings. Achievement of this goal on a sustainable basis required an improved internal management reporting 1. Introduction system. Prior approaches to the evaluation and analysis of SRCE courses have been de- Crafty men contemn studies; scribed in the papers by Bekic,´ et. al. 1998 Simple men admire them; and and Milinovic,´ et. al. 2000 . Wise men use them. B. Aim: long term measurement, and moni- (Sir Francis Bacon, ’Of Studies’, 1625) toring of the course activity with properly This academic research project Project SOCRA- “modeled”, sensibly designed, easily main- TES sprang out of a desire by one of the tained, and user friendly application. The authors to build an easily managed software- application would allow managers and ad- based system for measuring, monitoring, and ministrators to establish an unbiased, quan- improving the quality of course offerings, and titative i.e., statistical foundation for bet- for enhancing and reallocating scarce teaching ter decision making on the course content, r 1 resources via graphics-oriented, SAS Insti- course development, and teacher training tute decision maker software at the Univer- and assignment. 1 SAS is a registered trademark of SAS Institute Inc., NC, USA. 198 Measuring and Monitoring Movements in Course Interest C. Preview: this paper includes a brief presen- administrators and staff was to use this in- tation and overview of the research process formation to help shape management resource used to complete the “proof-of-concept” pha- and scheduling decisions related to the course se of Project SOCRATES as follows: Prob- content, faculty assignmenttraining, and new lem definition, questionnaire design, data course development. collection continuously for four years ,data The instrument/questionnaire was designed analyses univariate, bi-variate, and multi- after accessing information available on the In- variate, interim results evaluation, prelimi- ternet primarily from the University of Colo- nary analytical reporting results, and decisi- rado, Boulder and the University of Maryland, on-oriented, software application develop- European Division regarding methods for eva- ment. luating both course content and teacherpresen- D. Intended contribution: the goal of this re- ter attributes. Attributes, mostly satisfaction search is to help shape management decision ratings, please contact SRCE for a copy of the making behavior in an economical and rela- instrument were measured on both five 5 an- tively unbiased fashionso as to better al- chor point scales and binary type “yes” “no” locate scarce public resources in accordance scales. Various basic demographic informa- with student-community needs, and to iden- tion on the students and on SRCE facility was tify and either remedy, or else learn from also gathered using categories selected based “statistically unusual” coursepresenter sa- on Central Bureau of Statistics guidelines. This tisfaction ratings. allowed for a comparison between SOCRATES demographic results with Croatian students, ex- cluding staff and faculty in general an indi- Methodology rect method sometimes used for inferring the likelihood of having achieved “representative- ness” in the sample. In fact, SOCRATES re- Problem definition: The research problem was spondents were found to be not significantly defined as “an attempt to test whether courses different in age by number of years of ed- are evaluated as essentially equal, professors ucation from the general population of stu- rated as essentially equal, and demand for va- dents at large. That is, SRCE’s target users rious courses was essentially static” over the seem to reflect a representative cross section study period 4 years: 1996–2000 . During this of students by AGE. However, additional re- period, there was a total of 936 courses given, search is required to determine potential dif- evaluated by 9,249 students, representing a to- ferences that may exist across “areas of con- tal of 54,617 student-hours. Ultimately, the ob- centration” e.g., Arts, Mathematics, Electro- jective was to identify potential causes for any technicalComputing, Economics, etc. differences in courseteacher evaluation, and For purposes of this research, the universe was when changes were found, to establish “pro- defined as all students, staff and faculties that bable cause” for movements trends in demand were authorized to take SRCE courses. The for various courses as an aid to planning. sample frame would then logically consist of a The research objectives were to test the follow- list of names of everyone that took a course at ing three major hypotheses: SRCE during the time period under considera- tion. H01: All courses are being rated as essentially equal in terms of student’s overall satisfac- However, because all students did not complete tion; all courses, because of the difficulty in obtaining timely lists, and because of administrative costs, H02: All professors are being rated as essentially the actual respondents sampled for this research equal in the minds of students; represent a “convenience sample”. That is, all students that attended complete course were H03: The demand for courses and the course ra- tings are essentially static over time. asked, by the teacherpresenter at the end of the final class session, to fill in the course eva- Again, the business objective of the research luation form and return it to the instructor be- as was established in conjunction with SRCE fore leaving. Since the students were not tak- Measuring and Monitoring Movements in Course Interest 199 ing courses for grades, and since the teachers 2. It can be used to identify the groups i.e., expressed interest in improving the course con- courses, groups of courses, or teachers that tent and teaching skills, it was assumed that are rated significantly higher than average, the lack of total student anonymity did not se- and those that are rated significantly lower riously influence the students’ willingness to than average with ANOM one compares provide unbiased feedback on the course and group means against the overall mean.; on the teacherpresenter. 3. The graphical output is relatively self-expla- However, because of the nature of the sam- natory. pling design, the reader is cautioned regarding the wholesale adoption of the results presented Canonical discriminant analysis was applied in below in that they may be subject to “non- order to identify the key drivers of discrimina- observation”, non-sampling errors of unknown tion among the course or instructor ratings and magnitude. For example, students who either to position graphically individual courses or refused to cooperate in the research, or who instructors on the first few statistically signifi- abandoned the course prior to completion...may cant and most important canonical dimensions. have given the courseprofessor much lower Alternatively, logistic regression was used to or a much higher overall satisfaction ratings. identify what drove key “satisfaction measures” Based on this “non-probability design”, the chan- as defined below . ce of this having happened is unknownand unknowable. It means that the results are lim- The third hypothesis was tested using the tests ited to those who did respond to the survey for proportions and Cochran-Mantel-Haenszel CMH correlation statistics for testing for the n 7000 and cannot be generalized to the students who did not2. change between the last two years and the Cochran-Armitage test for trend over the four- Future changes in research design could in- year period. The analysis of structural changes clude provisions for sealed envelopes, absence was also performed, but in this paper the results of teachers from the room during survey com- are only shown in the graphical form using pletion, preprinted questionnaire completion in- comparative Pareto charts and the growth-share structions, and a centralized envelope dropcol- matrix bubble plots. lection