SJSU Annual Program Assessment Form Academic Year 2013-2014

SJSU Annual Program Assessment Form
Academic Year 2013-2014

Electronic copy of report is due June 1, 2014. Send to Undergraduate Studies (), with cc: to your college’s Associate Dean and college Assessment Facilitator. List of AFs is found at

Department:Psychology
Program:BA, BS
College: Social Sciences
Website:

_ Check here if your website addresses the University Learning Goals. <If so, please provide the link.>

Program Accreditation (if any): n/a
Contact Person and Email:Clifton Oyamot, Associate Chair

Date of Report: 05/29/14

Part A

List of Program Learning Outcomes (PLOs)

Psychology Bachelor of Arts

(1) Knowledge Base of Psychology

1.1 Students will be able to demonstrate familiarity with the major concepts, theoretical perspectives, empirical findings, and historical trends in psychology.

Example(s) of Demonstrated Learning: Performance on a Core Concepts Psychology Exam

(2) Research Methods in Psychology

2.1 Students will be able to design, implement, and communicate basic research methods in psychology, including research design, data analysis, and interpretations.

Example(s) of Demonstrated Learning: Quality of original research and accompanying reports developed in research methods classes (Psychology 120).

(3) Critical Thinking Skills

3.1 Students will be able to use critical and creative thinking, skeptical inquiry, and a scientific approach to address issues related to behavior and mental processes.

Example(s) of Demonstrated Learning: Performance on a Psychology Critical Thinking Exam; performance on Psychology 100W major paper

(4) Applications of Psychology

4.1 Students will be able to apply psychological principles to individual, interpersonal, group, and societal issues.

Example of Demonstrated Learning: Quality of papers that apply psychological concepts to individual, interpersonal, group, or societal issues.

(5) Values in Psychology

5.1 Students will value empirical evidence, tolerate ambiguity, act ethically, and recognize their role and responsibility as a member of society.

Example(s) of Demonstrated Learning: Completing an online workshop on ethical principles in human research (successful completion of learning modules and accompanying quizzes required before certificate is issued).

Psychology Bachelor of Science

Includes the same learning outcomes as the BA program, with the following addition:

(2) Research Methods in Psychology

2.2 Students will be able to design, implement, and communicate advanced research methods in two or more focused domains within psychology, including research design, data analysis, and interpretations within each domain.

Example(s) of Demonstrated Learning: Quality of original research and accompanying reports developed in advanced, specialized research methods classes (Psychology 121A – E).

PLOs were decided upon through extensive faculty discussion and finalized in Fall 2005. The PLOs are consistent with national disciplinary standards as they were selected from those outlined in the American Psychological Association (APA) Guidelines for the Undergraduate Major. Criteria for assessing levels of mastery continue to be developed with assessment of each PLO. Details for those criteria can be found in specific Annual Assessment Reports. Just a few examples of how student demonstrate learning of each PLO are listed above.

Map of PLOs to University Learning Goals (ULGs)

Psychology, BA & BS / ULG 1 (Specialized Knowledge) / ULG 2 (Broad Integrative Knowledge) / ULG 3 (Intellectual Skills) / ULG 4 (Applied Knowledge) / ULG 5 (Social & Global Responsibilities)
PLO 1: Knowledge Base of Psychology. 1.1 Students will be able to demonstrate familiarity with the major concepts, theoretical perspectives, empirical findings, and historical trends in psychology. / X
PLO 2: Research Methods in Psychology. 2.1 Students will be able to design, implement, and communicate basic research methods in psychology, including research design, data analysis, and interpretations. / X
PLO 3: Critical Thinking Skills. 3.1 Students will be able to use critical and creative thinking, skeptical inquiry, and a scientific approach to address issues related to behavior and mental processes. / X
PLO 4: Applications of Psychology. 4.1 Students will be able to apply psychological principles to individual, interpersonal, group, and societal issues. / X
PLO 5: Values in Psychology. 5.1 Students will value empirical evidence, tolerate ambiguity, act ethically, and recognize their role and responsibility as a member of society. / X

PLOs were mapped to the ULGs through discussions held by the Psychology department Assessment Committee.

Alignment – Matrix of PLOs to Courses

Program Learning Outcomes
1. Knowledge Base of Psychology / 2. Research Methods in Psychology / 3. Critical Thinking Skills / 4. Applications of Psychology / 5. Values in Psychology
General Psychology (Psyc 001) / I / I / I / I / I
Elementary Statistics (Stat 095) / I
Psychobiology (Psyc 030) / R / R / R / R / R
Writing Workshop (Psyc 100W) / R / R / I
Child Psychology (Psyc 102) / R / R / R / R
Abnormal Psychology (Psyc 110) / R / R / R / A / R
Research Methods (Psyc 120) / A / R / A
Cognition (Psyc 135) / R / R / I
Human Learning (Psyc 155) / A / A / R / R / R
Perception (Psyc 158) / A / A / R
Personality (Psyc 139) / A / R / R / R / R
Social (Psyc 154) / A / R / R / R / R
Tests & Measures (Psyc 117) / R / R
Intermediate Statistics (Stat 115) / R / R
Clinical Psychology (Psyc 160) / A / R / R / A / A
Neuroscience (Psyc 129) / A / A / R / A / R
I/O Psychology (Psyc 170) / R / R / R / R
Capstone Courses (Psyc 190/195) / A / A / A / A / A

I = Introduce:Provides a student with a beginning knowledge or first experience of something

R = Reinforce:Strengthens a student's knowledge by providing additional depth, material, or experience

A = Advanced:Provides students with a concentrated and highly developed exposure

Planning – Assessment Schedule

We are wrapping up our current 5 year cycle of assessments and have assessed each of our PLOs at least once.

We will be completing our department self-study in Fall 2014. During this process we will be determining our assessment schedule for the next 5 year. Generally, we assess 1 – 2 PLOs each semester.

Student Experience
PLOs are communicated to students in course syllabi and department website,

Student feedback was not formally considered when creating PLOs. However, over the last 2 years we have administered an exit survey for graduating seniors that assesses their (a) knowledge of the PLOs and (b) opinions about which courses in the curriculum best meet each PLO. This data, and other student input, will be used when we next revisit and reconsider our department PLOs.

Part B

Graduation Rates for Total, Non URM and URM students (per program and degree)

Fall 20051 Graduation Summary, “SJSU Approach”

Total

First-Time Freshmen / Undergraduate Transfers
4-Year Rate / 6-Year Rate / 2-Year Rate / 4-Year Rate
Num Entering / Major
Total / 57 / 52.6%
/ College
54.4%
/ Univ
70.2%
/ Major
57.9%
/ College
64.9%
/ Univ
86.0%
/ 12 / Num Entering / Major
Total / 111 / 54.1%
/ College
55.9%
/ Univ
56.8%
/ Major
72.1%
/ College
74.8%
/ Univ
77.5%

URM

First-Time Freshmen / Undergraduate Transfers
4-Year Rate / 6-Year Rate / 2-Year Rate / 4-Year Rate
Num Entering / Major
Total / 23 / 39.1%
/ College
47.8%
/ Univ
60.9%
/ Major
43.5%
/ College
60.9%
/ Univ
82.6%
/ 12 / Num Entering / Major
Total / 30 / 50.0%
/ College
50.0%
/ Univ
50.0%
/ Major
70.0%
/ College
70.0%
/ Univ
70.0%

Non-URM

First-Time Freshmen / Undergraduate Transfers
4-Year Rate / 6-Year Rate / 2-Year Rate / 4-Year Rate
Num Entering / Major
Total / 28 / 64.3%
/ College
64.3%
/ Univ
78.6%
/ Major
71.4%
/ College
71.4%
/ Univ
89.3%
/ 12 / Num Entering / Major
Total / 60 / 60.0%
/ College
61.7%
/ Univ
63.3%
/ Major
75.0%
/ College
76.7%
/ Univ
81.7%

1 first semester in which 6-year rates appear in reports

Headcounts of program majors and new students (per program and degree)

Enrollment by Gender, Concentration, and Degree Type

Fall 2009 / Fall 2010 / Fall 2011 / Fall 2012 / Fall 2013
F / M / Total / F / M / Total / F / M / Total / F / M / Total / F / M / Total
BA / 717 / 308 / 1,025 / 627 / 254 / 881 / 561 / 225 / 786 / 544 / 203 / 747 / 673 / 237 / 910
BS / 51 / 21 / 72 / 81 / 42 / 123 / 148 / 63 / 211 / 139 / 62 / 201 / 177 / 79 / 256
MA / 19 / 5 / 24 / 17 / 5 / 22 / 15 / 8 / 23 / 14 / 10 / 24 / 14 / 12 / 26
MIN / 1 / 1
Total / 787 / 334 / 1,121 / 725 / 301 / 1,026 / 724 / 296 / 1,020 / 697 / 276 / 973 / 864 / 328 / 1,192
Spring 2010 / Spring 2011 / Spring 2012 / Spring 2013 / Spring 2014
F / M / Total / F / M / Total / F / M / Total / F / M / Total / F / M / Total
BA / 636 / 280 / 916 / 543 / 234 / 777 / 509 / 201 / 710 / 520 / 174 / 694 / 640 / 213 / 853
BS / 60 / 26 / 86 / 86 / 47 / 133 / 130 / 59 / 189 / 121 / 61 / 182 / 169 / 72 / 241
MA / 16 / 4 / 20 / 14 / 3 / 17 / 15 / 8 / 23 / 12 / 9 / 21 / 14 / 11 / 25
Total / 712 / 310 / 1,022 / 643 / 284 / 927 / 654 / 268 / 922 / 653 / 244 / 897 / 823 / 296 / 1,119
Applied, Admitted & Enrolled byConcentration
Cohort Group :Department Total
PSYC - Psychology
TOT / Fall 2009 / Fall 2010 / Fall 2011 / Fall 2012 / Fall 2013
Applied Indicator / Admitted Indicator / Enrolled Indicator / Applied Indicator / Admitted Indicator / Enrolled Indicator / Applied Indicator / Admitted Indicator / Enrolled Indicator / Applied Indicator / Admitted Indicator / Enrolled Indicator / Applied Indicator / Admitted Indicator / Enrolled Indicator
Psychology / 1,795 / 1,001 / 247 / 1,918 / 971 / 234 / 1,914 / 1,259 / 309 / 2,096 / 1,112 / 254 / 2,795 / 1,945 / 481
Psychology/Clinical Psychology / 60 / 9 / 7 / 82 / 9 / 8 / 72 / 8 / 7
Psychology/Industrial and Org Psyc / 51 / 10 / 9 / 62 / 10 / 9 / 74 / 11 / 11 / 96 / 14 / 13 / 104 / 11 / 9
Total / 477 / 318 / 126 / 676 / 311 / 97 / 772 / 561 / 165 / 805 / 440 / 128 / 1,149 / 965 / 282
Total / 1,283 / 676 / 115 / 1,185 / 649 / 126 / 1,082 / 686 / 133 / 1,240 / 661 / 116 / 1,590 / 965 / 183
Total / 146 / 26 / 22 / 201 / 30 / 28 / 206 / 31 / 29 / 147 / 25 / 23 / 160 / 26 / 25
TOT / Spring 2010 / Spring 2011 / Spring 2012 / Spring 2013 / Spring 2014
Applied Indicator / Admitted Indicator / Enrolled Indicator / Applied Indicator / Admitted Indicator / Enrolled Indicator / Applied Indicator / Admitted Indicator / Enrolled Indicator / Applied Indicator / Admitted Indicator / Enrolled Indicator / Applied Indicator / Admitted Indicator / Enrolled Indicator
Psychology / 2 / 2 / 2 / 88 / 62 / 47 / 185 / 44 / 20 / 6 / 3 / 3 / 197 / 47 / 19
Psychology/Industrial and Org Psyc / 1 / 0 / 0
Total / 1 / 0 / 0
Total / 2 / 2 / 2 / 88 / 62 / 47 / 185 / 44 / 20 / 6 / 3 / 3 / 195 / 47 / 19
Total / 1 / 0 / 0 / 1 / 0 / 0

SFR and average section size (per program)

Student to Faculty Ratio (SFR)

Course Prefix: PSYC

SFR / Fall 2006 / Fall 2007 / Fall 2008 / Fall 2009 / Fall 2010 / Fall 2011 / Fall 2012 / Fall 2013
Lower Division / 45.2 / 53.4 / 42.8 / 31.9 / 34.9 / 46.3 / 51.5 / 42.1
Upper Division / 28.8 / 28.1 / 30.9 / 29.8 / 30.0 / 31.3 / 31.8 / 31.4
Graduate Division / 9.5 / 9.2 / 8.9 / 8.6 / 7.7 / 8.2 / 7.9 / 11.1
Total / 26.7 / 27.4 / 27.3 / 24.7 / 25.6 / 29.2 / 30.5 / 31.1
SFR / Spring 2007 / Spring 2008 / Spring 2009 / Spring 2010 / Spring 2011 / Spring 2012 / Spring 2013 / Spring 2014
Lower Division / 49.5 / 47.6 / 57.7 / 36.7 / 45.7 / 42.4 / 53.4 / 42.3
Upper Division / 25.5 / 25.7 / 27.6 / 26.4 / 28.1 / 29.2 / 30.5 / 32.6
Graduate Division / 6.7 / 8.7 / 7.8 / 8.3 / 8.7 / 8.0 / 6.7 / 10.1
Total / 23.8 / 25.7 / 26.8 / 24.4 / 27.5 / 27.7 / 28.4 / 31.6
Average Headcount per Section- Data Exhibit 2
Prefix PSYC - Psychology
Overall Total
Average Section Size / Fall 2003 / Fall 2004 / Fall 2005 / Fall 2006 / Fall 2007 / Fall 2008 / Fall 2009 / Fall 2010 / Fall 2011 / Fall 2012 / Fall 2013
Lower Division / 67.5 / 78.6 / 65.4 / 51.8 / 51.0 / 49.5 / 34.3 / 33.8 / 48.4 / 56.7 / 49.1
Upper Division / 30.2 / 31.1 / 32.6 / 30.8 / 31.7 / 29.5 / 30.0 / 25.8 / 33.0 / 32.4 / 34.8
Graduate Division / 9.0 / 13.3 / 7.8 / 6.8 / 7.1 / 8.0 / 6.8 / 6.5 / 5.9 / 6.7 / 6.9
All Level / 29.0 / 32.9 / 30.0 / 27.5 / 29.1 / 27.6 / 24.6 / 23.2 / 29.3 / 31.2 / 33.0
Lecture Courses (LEC)
Average Section Size / Fall 2003 / Fall 2004 / Fall 2005 / Fall 2006 / Fall 2007 / Fall 2008 / Fall 2009 / Fall 2010 / Fall 2011 / Fall 2012 / Fall 2013
Lower Division / 93.0 / 90.9 / 69.3 / 59.0 / 69.4 / 71.7 / 63.9 / 47.2 / 52.7 / 62.6 / 53.2
Upper Division / 42.6 / 45.8 / 43.0 / 38.9 / 42.7 / 42.1 / 46.9 / 39.9 / 43.7 / 44.2 / 42.2
All Level / 53.3 / 56.4 / 48.7 / 43.8 / 49.2 / 48.4 / 50.1 / 41.5 / 46.2 / 49.2 / 45.1
Seminar Courses (SEM)
Average Section Size / Fall 2003 / Fall 2004 / Fall 2005 / Fall 2006 / Fall 2007 / Fall 2008 / Fall 2009 / Fall 2010 / Fall 2011 / Fall 2012 / Fall 2013
Lower Division / 23.0 / 29.5 / 23.0 / 18.3 / 18.9 / 17.3 / 17.4 / 18.0 / 18.7 / 19.3 / 20.3
Upper Division / 30.0 / 28.2 / 25.4 / 24.4 / 26.4 / 29.6 / 30.1 / 27.1 / 26.2 / 26.4 / 26.2
Graduate Division / 14.3 / 15.9 / 11.4 / 11.4 / 11.8 / 12.9 / 11.4 / 10.2 / 10.3 / 13.0 / 16.3
All Level / 21.4 / 21.4 / 19.3 / 18.6 / 19.2 / 21.0 / 20.8 / 19.0 / 18.9 / 21.2 / 23.3
Supervision Courses (SUP)
Average Section Size / Fall 2003 / Fall 2004 / Fall 2005 / Fall 2006 / Fall 2007 / Fall 2008 / Fall 2009 / Fall 2010 / Fall 2011 / Fall 2012 / Fall 2013
Upper Division / 1.3 / 1.2 / 1.6 / 2.5 / 1.8 / 1.8 / 1.6 / 1.1 / 1.3 / 1.4 / 1.4
Graduate Division / 2.8 / 5.8 / 3.3 / 2.4 / 2.3 / 3.9 / 2.9 / 2.5 / 2.2 / 2.3 / 2.2
All Level / 2.0 / 2.5 / 2.6 / 2.5 / 2.1 / 2.8 / 2.2 / 1.6 / 1.8 / 1.9 / 1.9

Percentage of tenured/tenure-track instructional faculty (per department)

InstructionalFTEFbyTenure Status
Department of Psychology
FTEF / 2009/10 / 2010/11 / 2011/12 / 2012/13 / 2013/14
Fall 2009 / Spring 2010 / Avg / Fall 2010 / Spring 2011 / Avg / Fall 2011 / Spring 2012 / Avg / Fall 2012 / Spring 2013 / Avg / Fall 2013 / Spring 2014 / Avg
Tenured / 13.0 / 12.2 / 12.6 / 13.2 / 12.0 / 12.6 / 13.0 / 13.1 / 13.1 / 12.9 / 13.0 / 13.0 / 13.2 / 10.4 / 11.8
Probationary / 4.3 / 4.2 / 4.2 / 3.0 / 3.0 / 3.0 / 2.2 / 2.0 / 2.1 / 3.4 / 3.0 / 3.2 / 4.0 / 2.4 / 3.2
Temporary / 12.5 / 14.4 / 13.5 / 12.2 / 15.7 / 13.9 / 16.9 / 17.2 / 17.1 / 15.2 / 14.1 / 14.7 / 14.7 / 12.9 / 13.8
Total / 29.8 / 30.7 / 30.3 / 28.4 / 30.7 / 29.5 / 32.1 / 32.4 / 32.3 / 31.6 / 30.1 / 30.8 / 31.9 / 25.7 / 28.8

Fall 2013

Total FTEF = 31.9

Tenured & Probationary = 17.2

% tenured/tenure-track instructional faculty = 17.2/31.9 = 54%

Spring 2014

Total FTEF = 25.7

Tenured & Probationary = 12.8

% tenured/tenure-track instructional faculty = 12.8/25.7 = 50%

Part C

Closing the Loop/Recommended Actions
The primary recommended action is to build upon the foundation we have created during the past assessment cycle to implement sustainable, reliable, and valid assessment during the next cycle. During this past assessment cycle we have developed promising measures for long-term, systematic evaluation of our students’ learning outcomes. One example is the development of a Core Concepts Exam to assess PLO 1 (Knowledge Base). Development of this exam was done collaboratively across a representative swath of our faculty. It has been tested on one sample of students and is currently being revised. Another example is the use of the Psychological Critical Thinking Exam (PCTE). The assessment committee developed a valid and reliable rubric for scoring parts of the PCTE, and this rubric will serve as a template for larger adoption of the PCTE. A long-term plan could involve collecting longitudinal data on PCTE and Core Concepts Exam performance, starting at Psychology 001 and ending at Psychology 190.

We expect the next cycled of data collection using the revised measures will yield valid and reliable data that will give us insight into how we are doing as a program – what we are doing well and areas that we need to strengthen.

Assessment Data
Analysis
Proposed changes and goals (if any)

I elected to combine responses to items 11 – 13 into a single report. This year we completed assessment of PLO 3 (Critical Thinking) and PLO 5 (Values). See following pages for summaries.

Program Learning Outcome 5 Assessment Data, Analysis, and Proposes Changes

PLO 5. Students will value empirical evidence, tolerate ambiguity, act ethically, and recognize their role and responsibility as a member of society.

Specific aspect measured (from APA CyberGuide, 2009)

5.1 Recognize the necessity of ethical behavior in all aspects of the science and practice of psychology.

Primary Measure

Percentage of 120 students who successfully completed an online IRB workshop and received a certificate of completion through Collaborative Institutional Training Initiative (CITI) or the NIH Office of Extramural Research. The CITI workshop is the same IRB training required of all faculty and staff at SJSU who intend to conduct or sponsor research on human participations. Researchers complete a series of modules in which they learn about aspects of ethical conduct in research, and then must pass a test of that knowledge before moving on to the next module. Certification is awarded when all required modules have been successfully completed.

Methods

Sample

All Psychology 120 sections offered in Fall 2013 (N = 5, 3 instructors, total # students = 140).

Materials

Instructors completed a Research Ethics Assessment survey asking about their use of research ethics-related activities and evaluations. For this report I focused on the online IRB workshop questions.

Results

Is completion of an online IRB workshop part of your course? / 5/5 (100%) = Yes
How many students completed this assignment at a satisfactory level (completed entire online training and received a certificate)? / 157/163 (96%)

Conclusion

All students who complete Psychology 120 are given opportunities to learn about ethical behavior in social science research. Nearly all of our students demonstrate satisfactory understanding of research ethics, as reflected in the completion of the online workshop (96%). Many also complete an IRB application that is reviewed by the course instructor, and nearly all who do demonstrate satisfactory performance (>95%).

Limits and Suggestions

We only sampled sections offered in Fall 2013 and a more comprehensive sampling is needed to determine the generalizability of these results. Future Psychology 120 instructors should be encouraged to continue including assignments such as online IRB workshop completion or IRB application completion as part of their course. Research ethics are probably discussed in many other courses in our major, and surveying beyond Psychology 120 would test whether this is the case.

Program Learning Outcome 3 Assessment Data, Analysis, and Proposes Changes

Program Learning Outcome 3

Students will be able to use critical and creative thinking, skeptical inquiry, and a scientific approach to address issues related to behavior and mental processes.

Specific aspect to measure (from APA CyberGuide, 2009)

3.3 Use reasoning to recognize, develop, defend, and criticize arguments and other persuasive appeals.

Primary Measure

We used the Psychological Critical Thinking Exam PCTE; Lawson, 1999), which assesses students’ ability to evaluate whether a researcher’s conclusion is warranted given the design of his or her study. The PCTE has 14 different scenarios in which a researcher makes a claim based on his or her data. Students are asked to “read each item then state whether or not there is a problem with the person’s conclusion and explain the problem (if there is one).” Each scenario has a fundamental flaw in the researcher’s conclusion or reasoning (e.g., inappropriate conclusion due to a lack of control/comparison group)

For this first iteration we focused on developing a valid and reliable rubric for Scenario 4 of the PCTE (see appendix). Rubric development involved initial rubric development and tests for interrater reliability on a subsample of student response. The assessment committee than revised the rubric to be used in the full analysis. Student responses were scored using a 5-point scoring system (0 = no problem identified to 4 = Most critical problem(s) identified and fully elaborated) with higher scores indicating better performance.

Sample

The test was administered to all Psychology 120 sections offered in Fall 2013 (N = 5, 3 instructors) and one section of Psychology 121A (N = 172). Of these, 80 were randomly selected to be scored. Each response was assigned 2 raters and v3.0 of the rubric was used to score the responses (see appendix). Rating discrepancies were resolved by a third rater.

Results

6% / 4 / Most critical problem(s) identified and fully elaborated
26% / 3 / Most critical aspect(s) of the most critical problem(s) specified with some description or explanation
30% / 2 / Some aspect(s) of the most critical problem(s) specified in a brief or superficial manner.
21% / 1 / A problem recognized but not one that is critical
16% / 0 / No problem identified

84% of our students approached the scenario with a critical mind and did not simply accept the (erroneous) conclusion offered by the research in the scenario (score ≥ 1).
62% correctly identified the critical problem in the scenario (score ≥ 2)
21% approached the scenario with a critical mind and identified potential problems with the conclusion drawn by the researcher, but did not identify the fundamental problem (score = 1).
16% uncritically accepted the (incorrect) conclusion of the researcher (score = 0).

Tentative Conclusions

Conclusions must be very tentative, but the results of this initial assessment are promising. For this first assessment, the manner in which the PCTE was administered and its role in the course grade was left to the discretion of the instructors. Consequently, there was variability across sections that may have impacted student motivation. For example, for some the PCTE was embedded in the final exam and for others it was offered as extra credit.

That being said, overall 84% of our students approached the scenario with a critical mind and did not simply accept the (erroneous) conclusion offered by the research in the scenario, and 62% correctly identified the critical problem in the scenario (lack of control/comparison group).

Limits and Suggestions

Additional data collection is needed to draw more firm conclusions. An obvious next step is to administer the PCTE in a systematic fashion and in such a way as to maximize student motivation and ability to do well. To this end, we recommend that the PCTE be administered in 120 and 190 sections as part of a final exam, that the instructions be modified to indicate what a fully elaborated answer would consist of, and be given enough space for a lengthy response.

The rubric development process we created can be usefully applied to the other items in the PCTE. That is, the rubric we developed for scenario 4 can serve as a template for the other scenarios.

We may consider training psychology graduate students to rate undergraduate responses. This training would reinforce and deepen research design concepts for our graduate students. Furthermore, this would provide practical training and experience for those masters students who wish to go on to a PhD program and/or eventually teach.