Wpływ testu na środowisko edukacyjne
Dzięki współpracy redakcji JOwS z Cambridge English Language Assessment mamy przyjemność udostępnić Państwu na łamach naszego czasopisma artykuł Nicka Saville’a - jednego z czołowych badaczy, zajmujących się problematyką testowania biegłości językowej. Tekst opisuje model umożliwiający badanie, w jaki sposób ocenianie umiejętności językowych kształtuje konteksty edukacyjne. Tekst w języku angielskim jest poprzedzony krótką syntezą w języku polskim, opracowaną przez dr Agnieszkę Dryjańską, redaktor JOwS.
Applying a model for investigating the impact of language assessment within educational contexts: the Cambridge English approach
In Research Notes 42 (2010), I explained why Cambridge English Language Assessment as an international test provider needs a model to guide its work in investigating the impact of its examinations. In this article I set out some features of the model now being developed and explain how it can be applied in the case of the Cambridge English examinations. The operational practices needed to implement this approach are being introduced incrementally and are being adapted and revised in light of experiences in conducting projects which are now underway in many parts of the world.
Impact research within Cambridge English
Impact research investigates and seeks to understand the effects and consequences which result from the use of tests and examinations in educational contexts and throughout society. As a field of enquiry it appeared in the language testing literature as an extension of washback in the 1990s. (See Cheng, Watanabe and Curtis 2004 for an overview of washback.) The PhD theses of Wall (2005), Cheng (1997, 2005) and Green (2007) published in the Studies in Language Testing series, looked at different aspects of washback and extended the earlier work of Hughes (1989) and Bailey (1996). While these studies inevitably touched on considerations related to impact, none proposed a comprehensive model which would allow complex relationships to be examined across wider educational and societal contexts. This has been the aim of the team working in Cambridge English.
The origin of the Cambridge English approach dates back to the early 1990s and to the time when the current test development and validation strategies were first introduced. In those early stages, Bachman’s work was influential as he was the first to present impact as a ‘quality’ of a test which should be integrated within the overarching concept of test usefulness (Bachman and Palmer 1996). Following his lead, Cambridge English also introduced impact as one of the five essential qualities, which together with validity, reliability, practicality and quality comprise the VRIPQ features of a test.
By conceptualising impact within VRIPQ-based validation processes from the start, there was an explicit attempt to integrate impact research into routine procedures for accumulating validity evidence. Subsequent work on impact has been framed by these considerations and since the initial stage it has been recognised that a proactive approach is needed to achieve intended effects and consequences.
In 1996, Milanovic and Saville proposed an early model of test impact which was explicitly designed to meet the needs of Cambridge English. They proposed four maxims as follows:
- Maxim 1 PLAN
Use a rational and explicit approach to test development;
- Maxim 2 SUPPORT
Support stakeholders in the testing process;
- Maxim 3 COMMUNICATE
Provide comprehensive, useful and transparent information;
- Maxim 4 MONITOR and EVALUATE
Collect all relevant data and analyse as required.
These maxims were designed to capture key principles and to provide a basis for practical decision-making and action planning – and they still remain central to the Cambridge English approach today (see Section 4.4 in Cambridge English Principles of Good Practice: Quality Management and Validation in Language Assessment (2013).
Under Maxim 1, Cambridge English endeavours to develop systems and processes to plan effectively using a rational and explicit model for managing the test development processes in a cyclical and iterative way. It requires regular reviews and revisions to take place and for improvements to be made when necessary (Cambridge English 2013:18–22, Saville 2003:57–120).
Maxim 2 focuses on the requirement to support all the stakeholders involved in the processes associated with international examinations. This is an important aspect of the approach because examination systems only function effectively if all stakeholders collaborate to achieve the intended outcomes.
Maxim 3 focuses on the importance of developing appropriate communication systems and of providing essential information to the stakeholders (Cambridge English 2013:12–14).
Maxim 4 focuses on the essential research requirement to collect as much relevant data as possible and to carry out routine analyses as part of the iterative model (noted under Maxim 1). The nature of the data needed to investigate impact effectively and how it can be collected, analysed and interpreted under operational conditions has become an increasingly important part of the model in recent years.
Three major impact studies were also carried out between 1995 and 2004. Project 1 was the survey of the impact of IELTS (International English Language Testing System). This project helped conceptualise impact research including the design and validation of suitable instruments. Project 2 was the Italian Progetto Lingue 2000 impact study and was an application of the approach within a single macro educational context. These two projects are described in detail by Hawkey (2006). Project 3 was the Florence Language Learning Gains Project (FLLGP). Still within Italy, this project was an extension and re-application of the model within a single school context (i.e. at the micro level). It focused on individual stakeholders in one language teaching institution, namely teachers and learners preparing for a range of English language examinations at a prestigious language school in Florence. The complex relationships between assessment and learning/ teaching in a number of language classrooms, including the influence of the Cambridge English examinations, were examined against the wider educational and societal milieu in Italy. The micro level of detail, as well as the longitudinal nature of the project conducted over an academic year, were particularly relevant in this case (Saville 2009).
Based on an analysis of these projects, I have proposed a meta-framework designed to provide a more effective model for conducting impact research under operational conditions (Saville 2009). I suggest that by implementing this framework more systematically, ‘anticipated impacts’ can be achieved more effectively and well-motivated improvements to the examination systems can be identified and put into place. Aspects of this approach are represented in the impact studies reported in this issue and are focused on in the second part of this paper under the concept of impact by design.