Wpływ testu na środowisko edukacyjne
Dzięki współpracy redakcji JOwS z Cambridge English Language Assessment mamy przyjemność udostępnić Państwu na łamach naszego czasopisma artykuł Nicka Saville’a - jednego z czołowych badaczy, zajmujących się problematyką testowania biegłości językowej. Tekst opisuje model umożliwiający badanie, w jaki sposób ocenianie umiejętności językowych kształtuje konteksty edukacyjne. Tekst w języku angielskim jest poprzedzony krótką syntezą w języku polskim, opracowaną przez dr Agnieszkę Dryjańską, redaktor JOwS.
The concept of impact by design
Impact by design is a key feature of the expanded impact model. It starts from the premise that assessment systems should be designed from the outset with the potential to achieve positive impacts and takes an ex ante approach to anticipating the possible consequences of using the test in particular contexts.
In the final part of this paper, the following four points which are central to the model are discussed:
- test features (constructs and delivery systems);
- outcomes over time – the timeline;
- research methods and roles of researchers.
Test features (constructs and delivery systems)
Impact by design builds on Messick’s (1996) idea of achieving ‘validity by design as a basis for washback’. The importance of the rational model of test development and validation with its iterative cycles is a necessary condition for creating construct valid tests and for the development of successful systems to support them (cf. Maxim 1). Adequate specification and communication of the focal constructs is crucial for ensuring that the test is appropriate for its purpose and contexts of use and to counter threats to validity: construct underrepresentation and construct irrelevant variance (Messick 1996:252).
Insights from socio-cognitive theory underpin contemporary theories of communicative language ability, language acquisition and assessment (cf. the socio-cognitive model (Cambridge English 2013:25–27, Weir 2005)) and are also helpful in understanding how language learning and preparation for examinations takes place in formalised learning contexts, such as classrooms.
While appropriate construct representation is a necessary condition for achieving the anticipated outcomes, it is not sufficient and impact by design highlights the importance of designing and implementing assessment systems which explicitly incorporate considerations related to the social and educational contexts of learning/teaching and test use. This relates to the need for effective communication and collaboration with stakeholders, as noted in the original Maxims 2 and 3 and incorporated into the Principles of Good Practice, Section 2 (Cambridge English 2013).
Understanding the nature of context within educational systems and the roles of stakeholders in those contexts are clearly important considerations for Cambridge English – see Saville (2003:60) for a discussion of stakeholders.
It is now widely recognised that educational processes (see Figure 1) take place within complex systems with dynamical interplay between many sub-systems and ‘cultures’ and so an understanding of the roles of stakeholders as participants is a critical factor in bringing about intended changes (e.g. Fullan 1993, 1999, Thelen and Smith 1994, Van Geert 2007).
In conducting impact research the aim is to understand better the interplay between the macro and micro contexts within the society where the tests are being used and to determine which elements facilitate or hinder the desired outcomes. In general, diversity and variation increases as one moves from the general milieu within a country or region (the macro context) to specific schools and ultimately to the individual participants within classrooms (the multiple micro contexts at the local level involving schools, classes/groups and individual teachers and learners).
Figure 2 diagrammatically shows a school context embedded in a wider milieu with a teacher interacting with groups of learners in a particular classroom. The external influences include the general features of the milieu, as well as specific educational factors such the curriculum and syllabus and the need to produce examination results which are used outside of the school context.
It is therefore important to develop methods to understand both the general context as well as specific local cases, including dynamics which affect learning in classrooms. This points to the need to use both quantitative and qualitative data collection methods (see below).
In understanding the macro contexts into which international examinations are introduced (e.g. as part of educational reforms or innovations), it is important to focus on key factors related to the following:
- the political regime and its approach to educational reforms;
- the role of educational reforms within wider socioeconomic policies;
- cultural norms and expectations in relation to education generally, and attitudes towards language education (and towards English specifically in the case of Cambridge English);
- the educational system and how it is organised (e.g. compulsory education and the nature of the educational cycles; private vs. public schools; role of standardised assessment, etc.);
- broad differences between geographical regions and socioeconomic groups.
Collaboration between an international examination provider and local users is essential in order to capture relevant data and to shed light on such contextual parameters. Many dilemmas which arise in assessment contexts can only be dealt with if a wide range of local stakeholders agree to manage them in ways which they jointly find acceptable; the challenge is to get the relevant stakeholders working together effectively to agree what needs to be done to achieve the intended outcomes.
Outcomes over time – the timeline
It is essential to know what happens when a test is introduced into its intended contexts of use and this should constitute a long-term validation plan (cf. Maxims 1 and 4). Anticipating and managing change over time within specific contexts is therefore central to this concept and it means that appropriate consideration of timescales and the timeline for implementation (often involving several phases) are central to the design of impact studies. In impact research designs there is nearly always a fundamental need to collect comparative data, and therefore to develop research designs which can be carried out in several phases over an extended period of time or replicated in several different contexts.
Similarly, effects and consequences – intended and unintended – usually emerge over time given that contexts of use are not uniform and are subject to change, e.g. as a result of localised socio-political and other factors. Impact by design is therefore not strictly about prediction; a more appropriate term might be ‘anticipation’. In working with stakeholders, possible impacts on both micro and macro levels can be anticipated as part of the design and development process, and where potentially negative consequences are anticipated, remedial actions or mitigations can be planned well in advance.