CHANCELLOR’S OFFICE CALIFORNIA COMMUNITY COLLEGES STUDENT SERVICES AND SPECIAL SERVICES DIVISION REQUEST FOR APPLICATIONS
ASSESSMENT PSYCHOMETRIC TECHNICAL ASSISTANCE
JANUARY 30, 2013
APPLICATION DEADLINE: APPLICATIONS MUST BE RECEIVED AT THE CHANCELLOR’S OFFICE BY 5:00 P.M. ON MARCH 1, 2013 TABLE OF CONTENTS
PAGE
SECTION I REQUEST FOR APPLICATIONS (RFA) SPECIFICATIONS
Purpose...... 1 Background...... 1 Scope...... 2 Assessment Workgroup – Review Process ...... 2 Chancellor's Office Assessment and Evaluation Standards………………………………… 3 General Psychometric Consulting ...... 4 Funding…………………………………………………………………………………………….. 5 Minimum Qualifications...... 5 Calendar of Key Dates...... 6 RFA Contact Information…………………………………………………………………………. 6
SECTION II APPLICATION INSTRUCTIONS AND EVALUATION CRITERIA
Required Proposal Content – must address A-E
Application Instructions...... 8 A. Cover Letter and Contacts...... 8 B. Capacity...... 9 C. Budget...... 9 D. Project Management……………………………………………………………………. . 10 E. Sample of Relevant Work...... 10 Proposal Review...... 11 Application Review Process...... 11 Application Reviewers...... 12 Scoring Criteria...... 12 Notification of Intent to Award ...... 12 Appeal Process...... 13 RFA Clarification ...... 13 Terms and Conditions ...... 14 Reporting...... 14
SECTION III APPENDICES
APPENDIX A ASSESSMENT WORKGROUP CHARTER
APPENDIX B REQUEST FOR APPROVAL PROCESS – LOCAL COLLEGE SUMMARY
APPENDIX C PSYCHOMETRIC TECHNICAL ASSISTANCE STANDARD LEGAL TERMS AND CONDITIONS SECTION I
REQUEST FOR APPLICATIONS (RFA) SPECIFICATIONS SECTION I
CHANCELLOR’S OFFICE CALIFORNIA COMMUNITY COLLEGES
REQUEST FOR APPLICATIONS ASSESSMENT PSYCHOMETRIC TECHNICAL ASSISTANCE
PURPOSE
The California Community Colleges Chancellor’s Office (Chancellor's Office) is seeking a statistician with psychometric expertise and experience to review requests for assessment instrument approval, and review and revise the Standards, Policies, and Procedures for the Evaluation of Assessment Instruments Used in California Community Colleges (4th Edition, revised March 2001). The successful applicant will provide psychometric consulting and technical assistance to the Chancellor's Office on assessment topics that may include, but are not limited to, multiple measures implementation, common assessment, test development, and other assessment-related issues.
BACKGROUND
California Community College Regulations
Requires the Chancellor's Office to: Establish and update, at least annually, a list of approved assessment instruments and guidelines for their use by community college districts. These guidelines shall identify modifications of an assessment instrument or the procedures for its use which may be made in order to provide special accommodations required by section 55522 without separate approval by the Chancellor. Such guidelines shall also describe the procedure by which districts may seek to have assessment instruments approved and added to the list. The Chancellor shall ensure that all assessment instruments included on the list minimize or eliminate cultural or linguistic bias, are normed on the appropriate populations, yield valid and reliable information, identify the learning needs of students, make efficient use of student and staff time, and are otherwise consistent with the educational and psychological testing standards of the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education. Over the course of the last several years, the Chancellor's Office has relied on the California Community College Assessment Workgroup and statistician consultants to review assessment instruments, develop the Standards, Policies and Procedures for the Evaluation of
1 Section I-Request for Applications (RFA) Specifications
Assessment Instruments Used in the California Community Colleges (2001), referred to as the Assessment and Evaluation Standards, and to assist with the provision of technical assistance to local colleges and districts. The review process is a well-established process and the successful candidate will be expected to continue, while at the same time creating greater efficiencies.
SCOPE
ASSESSMENT WORKGROUP – REVIEW PROCESS
The assessment review process begins with a college or test vendor submitting evidence to the Chancellor's Office for review and approval. The type of evidence required is based on the following Chancellor's Office document, Standards, Policies and Procedures for the Evaluation of Assessment Instruments Used in the California Community Colleges (2001) (http://extranet.cccco.edu/Portals/1/SSSP/Matriculation/Assessment/ApprovedGuidelinesMarch2001.pdf).
The types of evidence submitted includes, but is not be limited to, validity, reliability, test bias, cut-score validity, disproportionate impact, and accommodations. Appendix B, Request for Approval Process – Local College Summary provides a synopsis of the process for local colleges. Consult the Standards, Policies and Procedures for the Evaluation of Assessment Instruments Used in the California Community Colleges (2001) for additional details and requirements for local colleges and test publishers. Evidence is reviewed by the Chancellor's Office and psychometric consultants. A preliminary report is written by the psychometric consultants and the submitting entity is provided an opportunity to respond to the preliminary report. Next, the Chancellor's Office Assessment Workgroup (Assessment Workgroup; see Appendix A for Assessment Workgroup charter) convenes and reviews all submissions, preliminary reports and responses, and provides recommendations for an appropriate approval level for the test. The psychometric consultants write the final report reflecting the Assessment Workgroup’s recommendations. The Chancellor's Office reviews the final report and all documents and makes a final determination on the appropriate approval level.
This is a biannual review process. The number of tests under review on any given cycle varies. For example, in spring 2012 the Chancellor’s Office received 24 tests from colleges and another 14 tests from two separate test publishers.
Based on the aforementioned review process, the Chancellor's Office compiles and publishes annually a list of approved assessment instruments for use in the California Community Colleges. Consult the “approved instruments” tab on assessment web-site for the list http://extranet.cccco.edu/Divisions/StudentServices/Matriculation/Assessment.aspx.
Assessment Workgroup services to be provided
The subcontractor will assist with the evaluation of test instruments used for placement in California Community Colleges and provide technical assistance to the Chancellor's Office and colleges on a variety of assessment-related topics. This work includes telephone or email consultations with local colleges and second-party test publishers about design, data collections, analysis procedures and the instrument review and approval process. It is
2 Section I-Request for Applications (RFA) Specifications
expected that the subcontractor will participate in planning and development meetings providing technical assistance on assessment, testing, and related topics to the Chancellor’s Office staff.
Specific tasks may include, but are not limited to:
1. Review initial or renewal approval request materials for locally developed/managed and second party assessment tests and project administration. This review process includes: a. provide initial review and analysis of all materials submitted b. prepare preliminary reports and recommendations for approval status and send to the local colleges after review and approval from the Chancellor's Office; c. review new material received from colleges or second party test vendors in response to the preliminary report and revision of reports d. submit revised reports and a summary of recommendations to the Assessment Workgroup for its meetings e. write revised final reports based on the Workgroup recommendations f. maintain all assessment files and reports g. assist with appeals proceedings h. attend Assessment Workgroup meetings i. assist in up-dating the information on the Chancellor’s list of approved tests for distribution
2. Conduct the following trainings/workshops on assessment, the test validation process and other assessment-related topics: a. Up to two (2) days of training for Chancellor's Office staff b. Up to three (3) days of training for the Assessment Workgroup members c. Provide training for community colleges and second-party publishers during a minimum of four statewide regional trainings (two in northern and two in southern California). The content and length of the trainings will be determined in collaboration with the Chancellor’s Office based on statewide need.
CHANCELLOR'S OFFICE ASSESSMENT AND EVALUATION STANDARDS
The Chancellor's Office Standards, Policies and Procedures for the Evaluation of Assessment Instruments Used in the California Community Colleges was last revised in 2001. The successful applicant will review and rewrite this document. This review will be done in consultation with the Chancellor's Office and appropriate stakeholders. The revised document will reflect current trends and technology in assessment and will incorporate strategies to make the assessment test review process more efficient.
It is estimated that approximately ten days will be required to update the Standards, Policies and Procedures for the Evaluation of Assessment Instruments Used in the California Community Colleges (2001).
3 Section I-Request for Applications (RFA) Specifications
GENERAL PSYCHOMETRIC CONSULTING
The Chancellor's Office may have the need for psychometric consulting in a broader context than the aforementioned Assessment Workgroup review process and Assessment and Evaluation Standards document update. This consulting may support the development of a statewide common assessment system, an ESL test, test development in general, multiple measures implementation, or pilot projects throughout the state. Examples of the types of projects that may fall under this category include:
Common Assessment System
In fall 2011 the Governor signed AB 743. This legislation requires the Board of Governor’s to establish a common assessment system and that the Chancellor's Office works in collaboration with the State Department of Education and the California State University when developing a common college-readiness standard that will be reflected in the creation of assessment instruments. In January 2012, the Student Success Task Force built upon the theme of common assessment and recommended that the common assessment be diagnostic. To date, the Chancellor's Office has convened the Centralized Assessment Advisory Committee to explore how to implement the requirements of AB 743. Items under consideration include developing a data warehouse, exploring the feasibility of a document to equate test scores from various tests, and completing the ESL Test.
English as a Second Language Test
Since 2007-08, the Chancellor's Office has funded efforts to develop an ESL test instrument for use by all community colleges. Thus far, two banks of ESL test-items have been developed: 1) Reading (232 test items) and 2) Language Structure (285 test items), written for three representative CCC ESL levels – Novice, Intermediate, Advanced. The Reading test items were field tested at colleges via a total of nine subtest forms -- three forms per level. The field tested data were analyzed using CTT and 94% of the 225 items (associated with the 45 reading passages) demonstrated strong reliability statistics The Chancellor's Office may require psychometric expertise to guide the ongoing development of this test.
Multiple Measures
Title 5 prohibits colleges from using only a single assessment test and requires the use of multiple measures. The Chancellor's Office continues to provide technical assistance to the field on the use of multiple measures in assessment and may require psychometric guidance in this area.
General Psychometric Consulting services to be provided
4 Section I-Request for Applications (RFA) Specifications
The subcontractor will provide general psychometric consultation to the Chancellor's Office. This consultation may include, but is not limited to, providing direction on the development of the ESL test that will lead to a computer-adaptive test such as specifying the number of test- items to develop and assisting with the development of an administrative plan and maintenance schedule. Psychometric consultation to the Chancellor's Office may cover multiple measures, cut-scores, common assessment, and other testing and assessment topics as they arise. Work may include participating on conference calls, attending meetings, writing documents, and answering questions from stakeholders.
The Chancellor's Office anticipates requiring up to seven (7) consultant days dedicated to the category of general psychometric support to the Chancellor's Office (excluding including the Assessment Workgroup assessment test review process and Assessment and Evaluation Standards document update).
FUNDING
Funding is for 18 months and is not to exceed $100,000. Funding for subsequent years is contingent upon Chancellor’s Office need, satisfactory performance, and availability of funds.
The successful applicant will be awarded a subcontract with a community college district acting as a fiscal intermediary of the Chancellor's Office.
MINIMUM QUALIFICATIONS
Whether you are an organization or an individual submitting an application, the minimum requirements include:
1. The principal consultant must have a Ph.D. in Educational Measurement, Psychometrics, Psychological Measurement, Statistics or related field 2. A minimum of five years of experience in the area of assessment 3. Test development experience 4. Research experience 5. Project planning and implementation experience Desired qualifications a. Exceptional communication skills b. Knowledge of California’s higher education assessment systems
c. Knowledge of computer-adaptive assessments and assessment management information systems
d. Awareness of and familiarity with the California Community College assessment review process and requirements
5 Section I-Request for Applications (RFA) Specifications
e. Knowledge of or familiarity with assessment practices in community colleges nationally, cut-score strategies, and the use of multiple measures
As an organization, these qualifications apply to the individual(s) who will be responsible for conducting the work under this subcontract (excluding clerical and technology support).
CALENDAR OF KEY DATES Key Dates Events January 30, 2013 RFA Released Information Session (webinar) TBD Information will be posted on the Chancellor's Office website: http://extranet.cccco.edu/Divisions/StudentServices/Matriculation.aspx#SB1456_Webinar_Oct_15,_2012 March 1, 2013 Application Deadline by 5 p.m. April 2-5 2013 Review Process April 15, 2013 Tentative Intent to Award and Notification, Website post April 25, 2013 Deadline to file appeal June 1, 2013 Subcontract finalized and work begins
RFA CONTACT INFORMATION
If you have any questions or need more information regarding this RFA, please contact RFA Administrator, Dr. Kimberly McDaniel by telephone (916) 323-0799, e-mail at [email protected], or by U.S. mail at California Community Colleges Chancellor’s Office, 4th Floor, 1102 Q Street, Sacramento, CA 95811.
6 SECTION II
APPLICATION INSTRUCTIONS AND EVALUATION CRITERIA
7 SECTION II
APPLICATION INSTRUCTIONS AND EVALUATION CRITERIA
APPLICATION INSTRUCTIONS
The following instructions prescribe the mandatory format and approach for the development and presentation of the application. The application instructions listed must be adhered to and all questions must be answered and all requested data must be supplied.
SUBMISSION
Submit 3 (three) copies of the application; two (2) hardcopies of which one (1) must contain original signatures (red or blue ink) on the Cover Letter. A third copy of the application must be submitted electronically.
Submit electronically in Word format to [email protected] by, Tuesday, March 1, 2013, by 5:00 p.m.
Please use the following naming conventions for submitting your application
Your organization name-Psychometric RFA
Your submittal must include the following attachments: CEO Letter [insert organization name] Application [insert organizational name] Sample of Relevant Work [insert organization name]
Please set the format for 8.5”x11” paper with margins of 1” in Arial font and 12-point type. A maximum of fifteen (15) pages will be accepted for your proposal only; this does not include the cover letter, the sample of relevant work, letters of support, or the budget.
Prepare your application with the following subheadings in bold 14-point font: Interest, Capacity, Minimum Qualifications, Budget, and Project Management Plan. Additional sub headers may be included.
A. Cover letter and contacts
The cover letter must be signed by the CEO or designee of the organization. Additionally, please provide:
8 Section II-Application Instructions and Evaluation Criteria
o Organization name, address, and main phone number o Organization CEO/Designee name o Principal contact information for this application (name, title, phone, and email).
B. Capacity (Maximum 125 points) Not to exceed 11 pages
B1. Describe organizational capacity: An organizational chart may be included (will not count towards the page-limit).
B2. Describe how the minimum qualifications provided on page five (5) are met. Successful applicants may address the organizations approach to, or experience with topics such as: o Assessment for course placement in higher education o Content validation, reliability, cut-scores, and all Chancellor's Office assessment standards http://extranet.cccco.edu/Portals/1/SSSP/Matriculation/Assessment/ApprovedGui delinesMarch2001.pdf o Steps of the test development process including item construction and initial and ongoing maintenance considerations o Common Core State Standards o Research methods o Test development o Test equating o Cut-score methodologies o Diagnostic assessments o Test administration o Computer-adaptive tests o Test administration technology and software o Holistic assessment o Multiple measures o National trends in community college assessment and placement practices
B3. Provide a brief description of how you will maintain appropriate documentation of the work completed and quality control for all deliverables, and describe the procedures for keeping files secure. Bear in mind that proprietary test-items, sensitive research, or confidential draft reports may be part of the materials you review.
9 Appendix A
C. Budget (Maximum 15 points) No page limit for the budget
NOTE: The purpose of the budget is to indicate whether the project is well planned and reasonable in scope. Technical errors in the budget will not necessarily result in a lower score, if minor assistance from the Chancellor’s Office can correct the error.
C1. The budget for this project will not exceed $100,000. Include the hourly rates, names, and titles for all staff who may be involved with this project. C2. Program funds are for direct services to the project only and are intended to supplement, not supplant existing organizational efforts. C3. An applicant may not propose to use funds to cover staff costs or to compensate outside individuals or firms for services associated with preparing this application. C4. The indirect administrative costs (overhead) for this project cannot exceed four percent (4%) of the total subcontract costs. Use the following formula where the hypothetical subcontract total is $300,000 to calculate maximum allowable indirect costs. Example: $300,000 - ($300,000/1.04) = $11,538
C5. The following are estimates to help guide the preparation of your budget. Please note these are only estimates. a. It is estimated to take between two to six hours to review initial or renewal materials. b. Approximately ten days will be required to update the Standards, Policies and Procedures for the Evaluation of Assessment Instruments Used in the California Community Colleges (2001). c. The Chancellor's Office anticipates requiring up to seven consultant days dedicated to the category of general psychometric support to the Chancellor's Office (excluding the Assessment Workgroup assessment test review process and rewriting the Chancellor's Office Assessment and Evaluation Standards). C6. Funds awarded under this RFA may not be used for purposes of construction or remodeling of facilities.
D. Project Management Plan (Maximum – 40 points) Not to exceed three (3) pages
a. Provide a management plan for operating the project. b. Show evidence of the commitment of project staff by describing their role and responsibilities (e.g., project manager, technology support, researcher, etc.) and the amount of time they will devote to project activities. Provide resumes/curriculum vitae for all professional staff who will work on this project. c. A project organizational chart may be included (will not count towards the page- limit).
10 Section II-Application Instructions and Evaluation Criteria
E. Sample of Relevant Work (Maximum 20 points) Preferably no more than 15 pages in length
Provide samples of written reports or other documents (do not include dissertation, thesis, or project(s) completed for school) related to the scope of this RFA. Examples could be correspondence, technical reports, articles, training materials, or any document(s) that demonstrate your ability to successfully comply with the RFA Specifications. More than one document may be submitted as long as the total number of pages submitted does not exceed 15.
Transition Plan
To ensure continuity of services, a transition plan will be required as part of the scope of work. This transition plan will be prepared after the award has been announced and finalized. The successful applicant in collaboration with the Chancellor's Office will address how project responsibilities will be transitioned without impact to project objectives. During plan preparation the applicant is encouraged to closely evaluate each responsibility with respect to how it will be resourced and what degree of transition is necessary. For example, in some instances a collaborative agreement with another entity may be the most effective resource alternative for a task.
The awardee will work closely with the existing subcontractors to ensure an uninterrupted transition of current program goals and commitments. The awardee should also maintain the current infrastructure necessary to maintain and provide secure access to current project and services.
PROPOSAL REVIEW
All applications will be judged against the same criteria in four (4) categories and scored on a 200-point scale. Each application will be rated on its own merits.
The Chancellor’s Office Reserves the Right to Reject Any and All Applications Received.
An application shall be rejected prior to scoring if:
1. It is received at any time after the exact time and date set for receipt of applications.
2. The application fails to meet the requirements of the RFA Specifications; or
3. The application contains false or misleading statements or references, which do not support an attribute or condition, contended by the applicant. The application shall be rejected if, in the opinion of the State, such information was intended to erroneously and fallaciously mislead the State in its evaluation of the application and the attribute, condition, or capability of a requirement of this RFA.
11 Appendix A
APPLICATION REVIEW PROCESS
This section is intended to help the applicants understand the processes and criteria that will be applied in determining which application is recommended for funding. The review is a holistic scoring process whereby the overall quality of each entire application is assessed and assigned a score that represents the best fit.
To ensure a fair and impartial award process, the Chancellor’s Office will perform the following procedures:
1. Conduct an initial screening of the applications to determine if they are in compliance with the RFA Specifications; application format; required signatures; and the number of copies. 2. Review the application to determine that the Minimum Qualifications are met. 3. Train and calibrate application reviewers. 4. Review applications. 5. Prepare and post notice of intent to award funds. This notice will be posted on the Chancellor’s Office Student Services Division and Special Programs web-site (http://extranet.cccco.edu/Divisions/StudentServices.aspx) no later than April 15, 2013.
APPLICATION REVIEWERS
The Vice Chancellor for Student Services and Special Services may select reviewers from members of the Chancellor’s staff, community college consultation groups and other entities that possess the appropriate expertise. All reviewers will receive intensive training and then review and score applications on the same designated day in 2013.
Individual teams of two trained reviewers will score every application independently then discuss the application. If the two scores are identical or if the difference of the two scores falls within a predetermined limit, the scores will be considered to be “in agreement,” and the two scores will be averaged. If the difference of the two reviewers’ ratings exceeds the limit, a third reviewer will re-score the application.
Please note that the Chancellor's Office may contact applicants as part of the application review process.
SCORING CRITERIA
All applications will be evaluated and ranked on the basis of the following:
1. Capacity...... 125 points 2. Budget...... 15 points 3. Project Management…………………...... 40 points
12 Section II-Application Instructions and Evaluation Criteria
4. Sample of Work...... 20 points Total…………………………….…………….... 200 points
Each category is reviewed and assigned a factor ranging from ‘Poor’ to ‘Exceptional’. Each factor is assigned a value range based on the maximum possible points for that category.
NOTIFICATION OF INTENT TO AWARD
Notifications of intent to award will be sent to the CEO of the organization indicating whether or not the application was approved. Chancellor’s Office staff will negotiate the specific scope of work, budget, and timeline for the awarded application. A public notification listing the tentative award will be posted in the Chancellor’s Office and Chancellor’s Office website on April 15, 2013.
Note: All questions regarding this notice for tentative awards should be addressed to Dr. Kimberly McDaniel (916) 323-0799 or [email protected] in the Student Services and Special Programs Division.
APPEAL PROCESS
Applicants who wish to appeal the Chancellor's Office decision must submit a letter of appeal to the Vice Chancellor, Student Services & Special Programs Division, Chancellor’s Office, California Community Colleges, 1102 Q Street 4th floor, Sacramento, California, 95814. Appeal letters must be hard copy and mailed by US Mail or other courier. The Chancellor's Office must receive the letter of appeal, with the original signature of the authorized person who signed the application within ten (10) days from the date of notification.
The letter of appeal must:
1. Be signed with an original signature by agency representative who signed the application.
2. Describe the factors that caused the applicant to conclude the prescribed rating standards were not followed.
3. Identify the specific information in the application that the applicant believes was overlooked or misinterpreted.
The Chancellor's Office will not consider incomplete or late appeals.
The Chancellor's Office staff or designees will reevaluate the application. The Vice Chancellor of Student Services and Special Services shall review all information submitted with regard to the appeal and render a decision regarding the protest within (30) calendar days. The decision of the Vice Chancellor shall be final.
RFA CLARIFICATION
13 Appendix A
If any ambiguity, conflict, discrepancy, omission, or other error in this RFA is discovered, immediately notify the Chancellor’s Office ([email protected]) of the error and request a written modification or clarification of the document. A clarifying addendum will be given to all parties who have obtained the RFA, without divulging the source of the request. Insofar as practical, the Chancellor’s Office will give such notice to other interested parties, but the Chancellor’s Office shall not be responsible for failure to do so.
TERMS AND CONDITIONS
Appendix B, Psychometric Technical Assistance Standard Legal Terms and Conditions, provides language that will be included in the executed subcontract with the successful candidate. It is included to highlight the copyright and intellectual property clauses.
REPORTING
Reporting requirements for the awardee will be decided in collaboration with the successful applicant and included in the executed subcontract. It is expected that the awardee will work closely with the assigned Chancellor's Office project manager.
14
APPENDIX A
APPENDIX A
CALIFORNIA COMMUNITY COLLEGES CHANCELLOR’S OFFICE ASSESSMENT WORKGROUP
Purpose of the Workgroup
The Statewide Assessment Work Group is a sub-committee of the Matriculation Advisory Committee (MAC). Members of this workgroup advise the Chancellor’s Office on statewide assessment issues, and conduct the biannual review of assessment instruments submitted by the colleges and test publishers for Chancellor’s Office approval.
Workgroup Composition
The committee is led by the Matriculation Program Coordinator of the Chancellor’s Office Student Services Division. The Assessment Workgroup is comprised of college assessment and research professionals, Matriculation Program Coordinators, and faculty. Psychometric consultation is a critical component to the success of the Assessment Workgroup. The number of committee members may vary from seven and shall not exceed 16. Membership of faculty is by Academic Senate appointment. Representation is reflective of constituent groups within the community college governance structure.
Given the technical nature of the Assessment Workgroup and the extensive training members receive upon joining the group, members may serve on the workgroup for up to six year with the understanding that Academic Senate appointments are one-year in length, but can be renewed. Renewal is encouraged by the Chancellor's Office.
Workgroup Structure
The committee will meet twice per year to review assessment validation submissions. The committee may also meet in person or via telecommunications alternative to address impromptu assessment issues.
The Chancellor's Office will develop meeting agendas with member input.
1 Appendix A
Anticipated Tasks
Review and evaluate assessment validation submissions twice per year and provide recommended levels of approval to the Chancellor's Office.
Assist the Chancellor's Office with the development of technical assistance materials on multiple measures and other assessment-related topics.
Provide guidance on the parameters guiding the implementation of AB 743 and assessment in general.
2 APPENDIX B
APPENDIX B
REQUEST FOR APPROVAL PROCESS – LOCAL COLLEGES SUMMARY
A. Guidelines Governing the Approval Process – The Chancellor's Office Standards, Policies and Procedures for the Evaluation of Assessment Instruments Used in the California Community Colleges (4th Edition, Revised March 2001)
B. Types of Evidence to be Submitted
1. Standard 1: Validity – Content, Criterion-Related, Consequential 2. Standard 2: Reliability – Test-Retest, Internal Consistency, Inter-Rater 3. Standard 3: Test Bias – Judgmental, Empirical 4. Standard 4: Cut Score Validity – Initial Setting of, Empirical Renewal Validation 5. Standard 5: Disproportionate Impact – Initial Plan, Empirical Renewal Data 6. Accommodations 7. Signed Chancellor's Office Request for Approval Form
C. Four levels of approval status exist.
1. A1: Full Approval (6 years) 2. A2: Provisional (one year + two years at Probationary level) 3. A3: Probationary (two years) 4. B: Not Approved (test is not included on list of approved tests)
D. Two minimums are required to attain an approval level.
1. There must be evidence of the test's validity for the intended purpose, and 2. There must be evidence that the test minimizes cultural/linguistic bias, insensitivity and offensiveness.
E. Initial approval is granted for a six-year period from the date that a test receives its first status in one of the approval categories. The approval status of a test must be renewed prior to the end of this six-year period in order for a college to continue its use for placement.
F. Using an approved commercially available second-party published test.
1 Appendix B
1. No information need be submitted (mailed) for review. Studies are to be conducted locally addressing three of the Review Standards: Content Validity, Cut-Score Validity and Disproportionate Impact.
2. If the results of these studies are supportive of use of the test, then the information is filed on campus and shared at the time of a site visit, or when requested by the Chancellor’s Office.
3. If the results of these studies are not supportive of use of the test, steps should be taken to rectify deficiencies in the placement system based on the data. This may include selecting a different test (e.g., content validity is not sufficient for all the courses), changing a test’s cut scores for placement into one or more of the courses, or investigating why disproportionate impact might be occurring, justifying its existence or taking steps to minimize it.
G. Using a locally managed/developed test.
1. Information must be submitted (mailed) for review addressing all five of the Review Standards: Content Validity, Reliability, Test Bias, Cut-Score Validity and Disproportionate Impact, plus a completed and signed CCCCO Request for Approval Form. Two cycles of review are conducted each year. Colleges may submit materials in mid-April (around the 15th) or in mid-November (around the 15th).
The information is to be sent by email or mailed to Dr. Kimberly McDaniel ([email protected]) at the Chancellor’s Office.
2. In order to be granted at least Probationary Approval status, the college must submit a minimum of one type of acceptable validity evidence and must satisfy the Test Bias standard.
3. During the “initial” submission for a test, either a judgmental or an empirical method may be used to establish cut-scores. Only a plan need be presented for monitoring Disproportionate Impact.
4. For “renewal,” some form of empirical data (consequential- or criterion-related validity) needs to be submitted to support the validity of the test’s cut-scores being used for placement. Data addressing disproportionate impact does need to be submitted when applying for approval renewal for an instrument. The data submitted must be from the prior three-year period.
5. Once materials are submitted, colleges receive a Preliminary Report of the initial review findings (within approximately one month). Based on these findings, the college may submit a response to clarify or supplement their materials (approximately three weeks are provided for a response).
2 Appendix B
6. If a response is received, the Preliminary Report is revised. The original/revised Preliminary Reports are distributed to members of the California Community College Chancellor’s Office Assessment Workgroup for their review and recommendations at meetings held bi-annually at the Chancellor’s Office (meetings held in January/early February for November submitted materials and in June for April submitted materials).
7. Based on the review and recommendations of the Assessment Workgroup, the Chancellor's Office makes a final recommendation and then the colleges are sent a Final Report identifying the approval level status for the test(s) submitted and the CCC Chancellor’s Office publishes the list of Approved tests (http://extranet.cccco.edu/Portals/1/SSSP/Matriculation/Assessment/ApprovedIns trumentsApril2009RevSept2011.pdf) for use as placement tools by California Community Colleges.
8. Colleges may appeal decisions made within 30 days of the receipt of the Final Report.
3 APPENDIX C
APPENDIX C
ASSESSMENT PSYCHOMETRIC TECHNICAL ASSISTANCE STANDARD LEGAL TERMS AND CONDITIONS
Copyright and Intellectual Property a. Subcontractor agrees that any and all services rendered and documents or other materials, inventions, processes, machines, manufactures, or compositions of matter, computer programs, computer software, and/or trademarks or servicemarks first created, developed or produced pursuant to this Agreement, whether by Subcontractor or others, shall be and are Work for Hire. All subcontracts shall include a Work for Hire provision by which all materials, procedures, processes, machines, computer programs, computer software, and trademarks or servicemarks produced as a result of this Agreement shall be Work for Hire. All rights, title, and interest in and to the Work first developed under this Agreement or under any subcontract shall be assigned and transferred to the Chancellor's Office. This Work for Hire agreement shall survive the expiration or early termination of this Agreement. b. The copyright for all materials first produced as a result of this Work for Hire agreement shall belong to the Chancellor's Office. Subcontractors and others that produce copyright materials pursuant to this Agreement, assigns all rights, title and interest, including the copyright to any and all works created pursuant to this Work for Hire agreement, to the Chancellor's Office. The Chancellor's Office shall acknowledge Subcontractors, if any, as the author of works produced pursuant to this Work for Hire agreement on all publications of such work. The Chancellor's Office may license Subcontractors to reproduce and disseminate copies of such work, provided the licensee agrees not to permit infringement of the copyright by any person, to compensate Chancellor's Office for any infringement which may occur, and to indemnify and hold harmless the Chancellor's Office for any and all claims arising out of or in connection with the licensing agreement. c. All materials first developed in draft and in final form pursuant to this Agreement shall, in a prominent place, bear the © (the letter "c" in a circle) or the word "Copyright," or the abbreviation "Copr.", followed by the year created; and the words "Chancellor's Office, California Community Colleges." Acknowledgment may be given to Contractor or the actual author(s) of the work in an appropriate manner elsewhere in the copyright material. If it is deemed necessary by either the Chancellor's Office or Subcontractor that the copyright be registered with the U.S. Copyright Office, Subcontractor will be responsible for applying for, paying the filing fees for, and securing said copyright. d. All technical communications and records originated or first prepared by Subcontractors, if any, pursuant to this Work for Hire agreement including papers, reports, charts, computer programs, and technical schematics and diagrams, and other documentation, but not including Subcontractor's administrative communications and records relating to this Agreement, shall be Appendix C
delivered to and shall become the exclusive property of the Chancellor's Office and may be copyrighted by the Chancellor's Office. e. If it is deemed necessary by either the Chancellor's Office or Subcontractor that a patent be obtained from the U.S. Patent and Trademark Office for any invention, process, machine, manufactures, or composition of matter, Subcontractor will be responsible for applying for, paying the filing fees for, and securing said patent. All patents for inventions, processes, machines, manufactures, or compositions of matter developed pursuant to this Agreement shall be issued to the "Chancellor's Office, California Community Colleges." All products and references to patents shall be marked and designated as such as required by law. Acknowledgment may be given to Subcontractor or the actual inventor(s) in an appropriate manner. The Chancellor's Office agrees to grant a nonexclusive license for such intellectual property to Subcontractor. Said license shall include the right to use the patent for inventions, processes, machines, manufactures, or compositions of matter derived from those created under this Agreement. f. All trademarks and servicemarks first created, developed or acquired pursuant to this Agreement shall be the property of the Chancellor's Office. If it is deemed necessary by either the Chancellor's Office or Subcontractor that a trademark or servicemark be registered with state or federal agencies, Subcontractor will be responsible for applying for, paying the filing fees for, and securing said protection. All trademarks and servicemarks obtained pursuant to this Agreement shall be issued to the "Chancellor's Office, California Community Colleges" and carry the designations permitted or required by law. The Chancellor's Office agrees to grant a nonexclusive license for the use of trademarks or servicemarks created, developed or obtained under this Agreement to Subcontractor. g. In connection with any license granted pursuant to the preceding paragraphs, Subcontractor agrees not to permit infringement by any person, to compensate Chancellor's Office for any infringement which may occur, and to indemnify and hold harmless the Chancellor's Office for any and all claims arising out of or in connection with such license. Subcontractor may, with the permission of the Chancellor's Office, enter into a written sublicensing agreement subject to these same conditions. h. Any and all services rendered, materials, inventions, processes, machines, manufactures, or compositions of matter, computer programs, computer software, and trademarks or servicemarks created, developed or produced pursuant to this Agreement by subcontractors that create works for this Agreement for Contractor are for and are the property of the Chancellor's Office. Contractor shall obtain an acknowledgement of the work for hire performed by these subcontractors that produce intellectual property pursuant to this Agreement, and all rights, title, and interests in such property shall be assigned to the Chancellor's Office from all subcontractors. Contractor shall incorporate the above applicable paragraphs, modified appropriately, into its agreements with subcontractors that create works for this Agreement. No unpaid volunteer or other person shall produce copyright materials under this Agreement without entering into a subcontract between such person(s) and Contractor giving the Chancellor's Office the foregoing rights in exchange for the payment of the sum of at least one dollar ($1).