FORUM ON ASSESSMENT ISSUES VII

“VALIDITY and RELIABILITY”

hosted by

PAMUKKALE UNIVERSITY School of Foreign Languages

November 25th, 2016

1

Dear FOAI-VII Participants,

It is a great pleasure to host the FOAI-VII event at Pamukkale University School of Foreign Languages. We hope it will be a fruitful platform by coming together, sharing our experiences, and learning from each other to gain deeper insight into issues regarding validity and reliability.

Like the first six events, we hope that this meeting will also enhance links between participants as well as create the opportunity to exchange ideas in order to improve assessment practices in our institutions. Please find below the PARTICIPANTS of FOAI VII – NOVEMBER 25th 2016.

Organizing Committee: Seher İşcan-Eda Duruk-Alev Özkardeş Döğüş

NAME INSTITUTION E-MAIL ADDRESS Selen Saydam Erdinç TOBB University of [email protected] Berkan Özkan and Technology [email protected] Zeynep Özcan Yaşar University [email protected] Esinç Demirkapu İstanbul Technical [email protected] Müge Cavcı University [email protected] Saba Sırt [email protected] Elif Aydın [email protected] Gülay Yiğit Koç University [email protected] İstanbul Şehir Mehtap İnce [email protected] University Işıl Alatan [email protected] Bilgi University Hande Günel [email protected] Canan Saban Özyeğin University [email protected] Merve Selçuk Kemerburgaz University [email protected] Seda Güven Eskişehir Osmangazi [email protected] Yeşim Yasemin Balık University [email protected] Pınar Aksoy Atılım University [email protected] Burcu Harputoğlu Fatih Sultan Mehmet [email protected] Çiğdem Sevilen Vakıf University [email protected]

Serkan Geridönmez [email protected] Fatma Soysal Adana University of [email protected] Ömer Gökhan Ulum Science and Technology [email protected] Özgür Ergül Pamukkale University [email protected] Ferhat Ordu Karabük University [email protected] Cenk Özdağ Doğu [email protected] Esin Çağlayan İzmir University of [email protected] Beatrice Yavaşer Economics [email protected] 2

RATIONALE/BACKGROUND

The Assessment Team of Sabancı University School of Languages set up the first FOAI forum for representatives from a range of universities in Turkey and North Cyprus to come together to discuss assessment issues in university foundation EAP/preparatory programmes. The forum aims to take a hands-on approach to the practical issues related to curriculum work, with a view to complement the formal presentation of papers or workshops to be found at other professional events such as conferences, seminars, and workshops. Although the forum was initiated by Sabancı University School of Languages, it has become a regular event, hosted by different institutions.

AIMS OF THE FORUM

The forum was set up specifically for representatives from the assessment teams of a range of foundation/preparatory programmes in English medium universities in Turkey. The forum aims to provide an opportunity for different state, foundation and private universities within the Turkish context to come together in order to:

. discuss issues concerning the design, development, and administration of low and high stakes exams; . share practices, experiences, ideas, and challenges faced relevant to testing, evaluation, and assessment related issues from diverse contexts; . increase awareness of the activities and approaches to assessment and evaluation of learning in other programmes; . build relationships and links between the assessment teams in different institutions; . exchange ideas on how members can cooperate in order to improve testing and assessment practices in their institutions.

THEME OF THE MEETING AND EXPECTED OUTCOME

The theme of the event is "Validity and Reliability” (as agreed on by the last FOAI participants). In the sessions, we intend to look into different practices of how placement and proficiency exams are carried out by considering whether it is valid and reliable, including the rationale for tasks that are used, and how their levels are set in each institution. This will involve the representative from each institution giving a short presentation (5-minutes) briefly outlining their practices. This will be followed by in-depth focus group discussions. Finally, the focus groups will give presentations based on their discussions which will be followed by a whole group discussion.

3

PROGRAMME Time Activity Place

Conference Hall 08.30 - 08.45 ARRIVAL & WELCOME (Ground Floor)

OPENING REMARKS Conference Hall 08.45 - 09.00 Assoc. Prof. Dr. Recep Şahin Arslan (Ground Floor) “How to ensure validity and reliability in prep schools”

Assoc. Prof. Dr. Turan Paker Conference Hall 09.00 – 09.15 “Preparing language-use items for A1, A2 and B1 learners” (Ground Floor)

Conference Hall 09.15 - 10.45 INITIAL MINI-PRESENTATIONS (Ground Floor)

10.45 - 11.00 COFFE-BREAK

FOCUS GROUPS Room 5,6 & 7 11.00 - 12.30 Focus group discussions and preparation of (Ground Floor) presentations based on discussion questions

Lunch 12.30 - 14.00 Food Court Group photo

FOCUS GROUPS CONTINUED Room 5, 6 & 7 14.00 - 14.30 Finalising presentations to whole group (Ground Floor)

FOCUS GROUPS PRESENTATIONS Conference Hall 14.30 – 16.00 Presentations to whole group (Ground Floor)

16:00 – 16:20 COFFE-BREAK

REFLECTION & LOOKING FORWARD Conference Hall 16:20 – 17:00 • Further issues for discussion (Ground Floor) • Date and host of the next event

4

INFORMATION ABOUT PRE-TASK FOR INITIAL MINI-PRESENTATIONS

Participants are asked to briefly describe the components of the placement and proficiency exams in their institutions in the initial presentations in the morning. Each institution will be allotted 3-5 minutes for the mini-presentation. Participants will be asked not to ask any follow-up questions at this point. Please note that the aim of this mini-presentation is to give other participants an initial idea about practices at your institution, and that you will have time during focus groups and later in the afternoon for further discussions. If you would like to use a PowerPoint presentation, you can use the template attached in this mail. It is advised that you limit your presentation to 3-4 slides.

Please send your PowerPoint presentation to [email protected] by Thursday, November 24.

INFORMATION ABOUT INITIAL MINI–PRESENTATIONS

AIMS

. To get to know a little about the other institutions represented at this forum

. To learn about other institutions’ practices

. To identify institutions which have dealt with similar issues with a view to learning more from them either during the focus group discussions or through subsequent follow-up contacts

FORMAT

Each institution will be allotted 3-5 minutes to briefly describe the procedures applied at their institution. No follow-up questions at this point. The aim of this presentation is to give other participants a brief introduction to the practices applied at your institution. Please note that the time limit will be strictly adhered to as more in-depth discussions will take place during the focus group discussions. The presenter may use PowerPoint (3-4 slides maximum) if they wish to do so.

INFORMATION ABOUT FOCUS GROUP DISCUSSIONS

After the initial mini-presentations, the participants will be divided into 3 groups to have an in- depth discussion on the issue at hand (See page 7 to see which group you are in). Each group will share their ideas and concerns (see aims below and discussion questions as guidelines) to help the discussion and a member of the host institution, a group facilitator, will be at hand to keep track of the discussion and take notes either directly into a PowerPoint document or to be transferred to PowerPoint later. At the end of the allotted time, the group participants are expected to present the main points of their discussions through PowerPoint to the rest.

5

AIMS

. To discuss in detail areas of common concerns regarding validity and reliability in exams

. To exchange ideas about how each institution has addressed these concerns; to share successes and failures in the related areas

. To brainstorm different suggestions on how to better respond to these concerns

. To summarise the highlights of the work of the focus group

Useful documents for the Focus Group Discussion

Participants are asked to bring samples of the following documents used at their institution as they could be used to assist the focus group discussions:

. soft copy of a Placement and Proficiency exam

. soft copy of any criteria / specifications / guidelines

. any other documents you feel may be relevant

FORMAT

We will be working in 3 separate groups. All groups will consider their ideas and concerns (see aims above as guidelines) regarding validity and reliability. It might be best for each group to spend five minutes or so at the beginning, to go through the questions and to prioritise the ideas to be discussed. During the focus group discussions, group facilitators will be taking notes and preparing a PowerPoint Presentation, which will be shared with all participants in the afternoon at the scheduled time. The presentation should be about 15 minutes long with 5 minutes for follow up questions/comments. The presentation will be e-mailed to all participants after the event. After each focus group presents, question and answer time will be allotted. Participants are also encouraged to reflect on similar concerns experienced at their institutions and share effective ways of addressing them.

REFLECTION

After the presentations and question-answer sessions are over, there will be a few minutes' reflection time on what was discussed throughout the day where participants will be encouraged to think about the following:

. One thing you may consider applying in your institution as a result of the discussions.

. One thing you would need to avoid or be more careful about when designing /selecting/administering/marking the Placement and/or Proficiency exam by considering its validity and reliability.

6

LOOKING FORWARD

After a few minutes of reflection, we will decide on the next host of FOAI-8 meeting, when it will take place, and brainstorm possible themes for the upcoming event.

FOCUS GROUPS

FOCUS GROUP A

Facilitator: Hülya Algan (Pamukkale University)

Participants: Canan Saban (Özyeğin University) Saba Sırt (Ege University) Işıl Alatan (Bilgi University) Selen Saydam Erdinç (TOBB) Fatma Soysal (Adana Science and Technology University) Yeşim Yasemin Balık (Eskişehir Osmangazi University) Müge Cavcı (İstanbul Technical University) Çiğdem Sevilen (Fatih Sultan Mehmet Vakıf University)

FOCUS GROUP B Facilitator: Nuri Tuna (Pamukkale University)

Participants: Serkan Geridönmez (Anadolu University) Burcu Harputoğlu (Fatih Sultan Mehmet Vakıf University) Zeynep Özcan (Yaşar University) Pınar Aksoy (Atılım University) Seda Güven (Eskişehir Osmangazi University) Cenk Özdağ (Doğu Akdeniz University) Elif Aydın (Ege University) Ömer Gökhan Ulum (Adana Science and Technology University) Esin Çağlayan (İzmir University of Economics)

7

FOCUS GROUP C Facilitator: İlker Bozbey (Pamukkale University)

Participants: Mehtap İnce (İstanbul Şehir University) Gülay Yiğit (Koc University) Esinç Demirkapu (İstanbul Technical University) Merve Selçuk (Kemerburgaz University) Ferhat Ordu (Karabük University) Özgür Ergül (Pamukkale University) Berkan Özkan (TOBB) Hande Günel (Bilgi University) Beatrice Yavaşer (İzmir University of Economics)

The following questions can be used to guide the focus group discussions.

Discussion Questions

1. Do you think your colleagues in your institution have enough background on validity/reliability? Are they familiar with the types/threats of validity and reliability?

2. Have you ever introduced these terms in workshops or teacher training programs in your institution?

3. What factors influence validity and reliability? How can we know that our exams are valid/reliable?

4. Is it valid when it is reliable or is it reliable when it is valid?

5. Inter-rater or Intra-rater reliability?

6. What are the practicality issues in speaking tests in your institution?

7. How do you enhance the following validity types in your institutions;-construct validity-content validity- response validity-concurrent validity-predictive validity-face validity

8. Scoring issues: How do you score writing and speaking tests? How is inter-rater reliability provided in them?

9. Which one should be preferred? Machine or human scoring?

10. Are there any institutions that use machine scoring? What is the process?

11. How do you ensure inter-rater reliability? Is there standardization?

12. Do you use Cronbach’s alpha/Kuder Richardson to measure reliability?

13. How long are your tests? How do you decide if the items are too easy/difficult? How do you ensure homogeneity?

14. How is backwash effect? How do your tests impact your teaching/learning? Do you think assessment leads learning or vice versa?

8

TRANSPORTATION INFORMATION

Pamukkale University Kınıklı Campus (Labelled on Google Maps as: Pamukkale Üniversitesi Yabancı Diller Yüksekokulu)

Address: Çamlaraltı Mh. Üniversite Cad. No:11/A Pamukkale/DENİZLİ

Telephone: 0(258) 296 20 00

1. From Çardak Airport:

A) Pamukkale Taxi: A fast option to come directly to Pamukkale University or to your hotel is to take the airport taxi (Pamukkale taxi). * If you let them know that you want to share the taxi with other passengers, it costs 30TL. per person. You can make a reservation in advance on 0537 305 45 45.

Duration: approximately 60 minutes

B) BAY-TUR Shuttle bus + Minibus You can take BAY-TUR shuttle busses (13tl) to go to Çınar (city center) and then take the minibus (2tl) to Pamukkale University.

Duration: approximately 90 minutes

2. From Bus Terminal:

Due to the improvement works on the main arteries, the most convenient way of transportation from the bus terminal is taxi (20-25tl).

Duration: approximately 15-20 minutes

Accomodation: 1. Şiir Hotel : 15 minutes walk from the main entrance of Pamukkale University Campus. 2. Polisevi : Opposite to the main entrance of Pamukkale University Campus. 3. Lavin Hotel : 15 minutes drive by taxi. 4. İnci Hotel : 10 minutes drive by taxi. 5. Hotels in Pamukkale (20 kms far away from the city center)

9