Course Assessment Report

COP 4020: Programming Languages I (Fall 2007)

Course Learning Outcomes and Expected Performance Criteria:

Outcome 1 [Concepts]: Use appropriate terms and concepts to explain the semantics of each essential programming model, be able to apply appropriate terms and concepts to answer questions about the meaning of small programs written in each model, and use appropriate terms and concepts in discussions of programming languages, so that at least 75% of the time these terms and concepts are used and understood correctly. Performance Criteria: 75% correct score on exam questions tagged with this outcome.

Outcome 2 [UseModels]: Use each essential programming model as realized in the Oz programming language to correctly solve at least 75% of small problems, when told what programming model to use. Performance Criteria: 75% score on exam questions tagged with this outcome.

Outcome 3 [UseLanguages]: Use an essential programming model in C, C++, or Java, to correctly solve at least 75% of small problems restricting your solution to a given programming model. Performance Criteria: 75% score on exam questions tagged with this outcome.

Outcome 4 [EvaluateModels]: For each essential programming model convincingly evaluate its advantages and disadvantages for solving a given programming problem (to the extent that your answer is convincing 75% of the time) Performance Criteria: 75% score on exam questions tagged with this outcome.

Outcome 5 [MapToLanguages]: Convincingly explain what features in C, C++, and Java support each essential programming model (to the extent that your answer is convincing 75% of the time). Performance Criteria: 75% score on homework problems and exam questions tagged with this outcome.

Relationship of the course to the Degree Program Outcomes:

• BSCS Degree Program Outcome #3: All graduating CS majors shall demonstrate knowledge of programming fundamentals by their use of programming languages. (addressed by course outcomes #1-5) (addresses CAC (i), (k)) • BSCS Degree Program Outcome #4: All graduating CS majors shall demonstrate their knowledge and understanding of, and their ability to apply, the concepts, design principles and fundamental algorithms relating to: data structures and their manipulation, programming languages, computer architecture and organization, computer operating systems, and computer networks. (addressed by course outcomes #1-5) (addresses CAC (a), (b), (c), (i), (j), (k)) • BCS Degree Program Outcome #6: All graduating CS majors shall demonstrate knowledge of fundamental concepts and principles in software engineering and their skill applying best- practices in software development processes, methods and tools. (addressed by course outcomes #1-5) (addresses CAC (a), (b), (c), (i), (j), (k))

CAC Outcomes for Computer Science The program enables students to achieve, by the time of graduation: a) An ability to apply knowledge of computing and mathematics appropriate to the discipline; b) An ability to analyze a problem, and identify and define the computing requirements appropriate to its solution; c) An ability to design, implement and evaluate a computer-based system, process, component, or program to meet desired needs; d) An ability to function effectively on teams to accomplish a common goal; e) An understanding of professional, ethical, legal, security, and social issues and responsibilities; f) An ability to communicate effectively with a range of audiences; g) An ability to analyze the local and global impact of computing on individuals, organizations and society; h) Recognition of the need for, and an ability to engage in, continuing professional development; i) An ability to use current techniques, skills, and tools necessary for computing practices. j) An ability to apply mathematical foundations, algorithmic principles, and computer science theory in the modeling and design of computer-based systems in a way that demonstrates comprehension of the tradeoffs involved in design choices; k) An ability to apply design and development principles in the construction of software systems of varying complexity.

Relationship of the course to CAC Outcomes:

Course CAC Outcomes Outcomes a b c d e f g h i j k 1 x x x x x x 2 x x x x x x 3 x x x x x x 4 x x x x x x 5 x x x x x x

Course Outcome to Program Outcome Mapping - Computer Science COP 4020 - Programming Languages I % of Course Students Outcomes Meeting CS CS CS CS CS CS CS CS CS CS Threshold #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 1 39.0% 39.0% 39.0% 2 56.1% 56.1% 56.1% 3 80.5% 80.5% 80.5% 4 85.4% 85.4% 85.4% 5 48.8% 48.8% 48.8% # greater than 50%/# 3/5 3/5

Course Content and Assessment Plan: Note that the homework assignments, although designed to teach the outcomes, are not normally part of the assessment of those outcomes, except for outcomes 3 and 5.

Course Degree Outcome# Program Assignments Covered Outcome# CAC Date Lecture Topic Description Due Covered Outcome Aug. 20 Introduction Aug. 22 Introduction to Programming Concepts Aug. 24 Introduction to Programming Concepts Aug. 27 Introduction to Programming HW 1 (1, 2), 5 3,6 a, b, c, i, j, k Concepts Aug. 27 Declarative Computation Model Aug. 29 Declarative Computation Model Aug. 31 Declarative Computation Model Sep. 3 Labor Day, no class Sep. 5 Declarative Computation Model Sep. 7 Declarative Computation Model Sep. 10 Declarative Computation Model Sep. 12 Declarative Computation Model Sep. 14 Declarative Computation HW 2, p. 1-4 (1,) 5 3,6 a, b, c, i, j, k Model Sep. 17 Declarative Computation Model Sep. 19 Declarative Computation HW 2, p. 5-10 (1, 2) 3,6 a, b, c, i, j, k Model Sep. 21 Declarative Programming Techniques Sep. 24 Declarative Programming Techniques Sep. 26 Declarative Programming Techniques Sep. 28 Declarative Programming HW 3, p. 1-6 (2) 3, 6 a, b, c, i, j, k Techniques Oct. 1 Exam 1 1, 2 3,6 a, b, c, i, j, k Oct. 3 Declarative Programming Techniques Oct. 5 Declarative Programming HW 3, p. 7-11 (2) 3, 6 a, b, c, i, j, k Techniques Oct. 8 Declarative Programming Techniques Oct. 10 Declarative Programming Techniques Oct. 12 Declarative Programming HW 3, p. 12-24 (2) 3, 6 a, b, c, i, j, k Techniques Oct. 15 Declarative Programming Techniques Oct. 17 Declarative Concurrency Oct. 19 Declarative Concurrency Oct. 22 Declarative Concurrency Oct. 24 Exam 2 1, 2 3,6 a, b, c, i, j, k Oct. 26 Declarative Concurrency Oct. 29 Declarative Concurrency HW 4, p. 1,3-5 (1, 2,) 3, 5 3,6 a, b, c, i, j, k Oct. 31 Declarative Concurrency Nov. 2 Declarative Concurrency Nov. 5 Declarative Concurrency HW 4, p. 7-21 (1, 2, 4) 3,6 a, b, c, i, j, k Nov. 7 Message Passing Nov. 9 Message Passing Nov. 12 Veteran's Day, no class Nov. 14 Message Passing Nov. 16 Exam 3 1, 2, 5 3,6 a, b, c, i, j, k Nov. 19 Message Passing HW 5, p. 1-2,4- (1, 2,) 5 3,6 a, b, c, i, j, k 6 Nov. 21 Relational Programming Nov. 23 Thanksgiving Break, no class Nov. 26 Relational Programming HW 5, p. 7,9 (1, 2,) 5 3,6 a, b, c, i, j, k Nov. 28 Relational Programming Nov. 30 Relational Programming Dec. 3 Course Summary and HW 6 (1, 2, 4,) 5 3,6 a, b, c, i, j, k Evaluation Dec 10 Final Exam 1, 2, 4 3,6 a, b, c, i, j, k

Assessment Result Details:

Measured Performance of Each Student on Each Course Learning Outcome (expressed as % of maximum possible, i.e. score out of 100 possible pts)

Course Learning Outcomes #1 through #5 [Use [Use [Evaluate [Map To Student [Concepts] Models] Languages] Models] Languages] Degree Student Code # Program #1 #2 #3 #4 #5 1 CS 78% 89% 100% 100% 94% 2 CS 86% 93% 100% 100% 80% 3 CS 94% 89% 100% 96% 87% 4 CS 58% 84% 100% 84% 89% 5 CS 59% 63% 77% 88% 77% 6 CS 47% 54% 0% 100% 25% 7 CS 96% 99% 100% 100% 70% 8 CS 75% 83% 100% 58% 93% 9 CS 61% 54% 100% 100% 58% 10 CS 62% 47% 38% 92% 79% 11 CS 65% 96% 90% 82% 62% 12 CS 58% 58% 92% 86% 80% 13 CS 71% 82% 100% 84% 82% 14 CS 68% 78% 100% 80% 66% 15 CS 83% 67% 100% 88% 80% 16 CS 76% 87% 23% 90% 68% 17 CS 74% 68% 14% 98% 55% 18 CS 78% 72% 90% 100% 49% 19 CS 75% 82% 83% 74% 60% 20 CS 70% 88% 100% 84% 44% 21 CS 74% 88% 100% 78% 80% 22 CS 100% 94% 100% 88% 92% 23 CS 86% 86% 0% 100% 54% 24 CS 45% 28% 77% 50% 72% 25 CS 82% 71% 92% 76% 78% 26 CS 65% 62% 95% 100% 57% 27 CS 82% 73% 77% 64% 74% 28 CS 68% 91% 95% 96% 92% 29 CS 73% 71% 95% 90% 76% 30 CS 70% 69% 80% 92% 79% 31 CS 67% 64% 0% 92% 44% 32 CS 85% 64% 100% 70% 68% 33 CS 87% 93% 100% 88% 88% 34 CS 55% 72% 73% 76% 73% 35 CS 60% 52% 95% 72% 61% 36 CS 88% 81% 92% 100% 90% 37 CS 75% 88% 90% 88% 80% 38 CS 85% 89% 100% 90% 92% 39 CS 74% 85% 55% 82% 42% 40 CS 65% 93% 100% 98% 57% 41 CS 91% 93% 100% 100% 68%

Course Learning Outcomes #1 through #5

Course Statistics #1 #2 #3 #4 #5 Mean 73.4% 76.6% 81.1% 87.2% 71.1% High score 100.0% 99.4% 100.0% 100.0% 94.1% Low score 44.5% 27.8% 0.0% 50.0% 25.1% Standard Deviation 12.9% 15.9% 30.8% 12.3% 16.3%

Course Learning Outcomes #1 through #5

Measurements vs. Criteria #1 #2 #3 #4 #5 Expected Performance Criteria 0.75% 0.75% 0.75% 0.75% 0.75% (expressed as % of maximum score out of 100 possible pts)

Number of Students 25 18 8 6 21 Below Criteria

Percent of Students 61.0% 43.9% 19.5% 14.6% 51.2% Below Criteria

Number of Students Meeting 16 23 33 35 20 or Exceeding Criteria Percent of Students Meeting 39.0% 56.1% 80.5% 85.4% 48.8% or Exceeding Criteria

Assessment Results Summary:

Results Summary of Course Learning Outcome 1 [Concepts]: The expected performance criteria was 75%, which only 39% of the students met. The assessment was based on exam questions. This evaluation is complicated, however, because in exam 2, several students did poorly on the relevant questions. We had a makeup exam for this, which not all students took, but in which most did better on these questions. If we take out exam 2 from this assessment, we find that 56.1% of the students meet the criteria. Still this is disappointing, and we’ll have to emphasize learning of concepts more in future offerings. I will be thinking about exactly how to do this.

Results Summary of Course Learning Outcome 2 [UseModels]: The expected performance criteria was 75%, which only 56.1% of the students met. This was something that we devoted a lot of class time and effort on, but some students still have done poorly on. I attribute this to the small amount of practice students had for the relevant problems on chapters 5 and 9, as if we exclude the test questions from exam 4 that tested these chapters (on message passing and the relational model), then 70.7% of the students meet the criteria for this outcome. I will redesign the homeworks for chapters 5 and 9 to give more relevant practice (smaller problems, and not very large problems that are too hard), which will bring the success for this outcome up for chapters 5 and 9 also.

Results Summary of Course Learning Outcome 3 [UseLanguages]: The expected performance criteria was 75%, which 80.5% of the students met. However, this assessment is based on only one homework problem (HW4, problem 4) and no exam questions. I found that it’s really not practical to be teaching students new languages and to also give significant exercises in Java (let alone in C++, which most of the students don’t know). So this outcome will be dropped in future offerings of the course.

Results Summary of Course Learning Outcome 4 [EvaluateLanguages]: The expected performance criteria was 75%, which 85.4% of the students met. This is good, although one should exercise caution, as the assessment is only based on a small number of exam questions.

Results Summary of Course Learning Outcome 5 [MapToLanguages] : The expected performance criteria was 75%, which 48.8% of the students met. This is based on a small number of homework and exam questions (weighted 40% for the homework and 60% for the exams, similar to the course weighting). Again, the small number of assessment points should give rise to caution in evaluating these numbers. However, it seems clear that the course should more actively stimulate student learning of how to map the concepts being learned to languages that they know. Perhaps improving this outcome can help students do better with the first outcome as well.

Course Recommendations based on Assessment Results:

The results show poor performance overall on outcomes 1 and 5, related to concepts. Clearly there is some mismatch here between what is being taught about the concepts, what the students are learning, and how the concepts are being assessed. Material that was problematic on exams 1 and 2 (related to free and bound variables and desugaring) needs to be strengthened, as it the material on free and bound variables doesn’t appear very clearly in the book and students are confused by the notion of “bound variable” (defined in a given scope) and “bound store variable” (a cell that has a value). The students seemed to understand the concept of desugaring later, but only slowly, so they need more practice with it, or it needs to be assessed later. But in general, I need to think about how to bring out both the concepts and the relation to existing languages more clearly. I think improving students’ understanding of how concepts map to existing languages can help the understanding of the concepts themselves, and will try to do more of that.

The results for outcome 2 “Use Models” are fairly good for the first part of the course (chapters 1-4, covering an overview of different models, functional programming, lazy functional programming, and declarative concurrent programming). This material has well-developed homework problems of appropriate size. However, during the course I discovered that my handout on “following the grammar” wasn’t sufficient for getting across the idea that each nonterminal in a grammar should correspond to a separate function (many students were using a single function for all the nonterminals). I created an example that shows students why this is a problem, and that seemed to help some of them, so I think this will help bring up the scores in the next semester for this part. For the material related to chapters 5 and 9, the homeworks didn’t have enough problems of an appropriate size to adequately teach this material. This will be corrected next semester, where I’ll add several problems to the homework for these chapter that are similar in size to those used in chapters 1-4, and abandon the 2 large problems that students complained about and didn’t do because they were too hard (without adequate smaller problems) and because it was too late in the semester for such large problems.

Outcome 4 “Evaluate Models” seemed to go well, so only incremental improvements are planned for it.

I plan to abandon outcome 3, of “Use Languages”, namely using C++ and Java in this course. There simply isn’t room for it and it’s not critical to the course objectives, which is to help students learn new ways of thinking about programming and new ideas in programming languages. Prepared by: Gary T. Leavens Date: 14-12-2007. revised 18-05-2008