An initial validity argument for a new B2 CEFR-related baccalaureate listening test

Resumen

Within the EU, governments and education departments have been obliged to take the CEFR into account, and consequently new educational initiatives have been plentiful as policy makers attempt to incorporate competence-based language education into state systems (Lim, 2013). In Spain, however, we are only just starting to see the influence of the CEFR. The recently revamped Spanish national curriculum Ley Orgánica para la Mejora de la Calidad Educativa (LOMCE) made law by the Ministry of Education, Culture & Sports (Boletín Oficial Del Estado (BOE), 2013); provides a new syllabus for English and has been elaborated using a competence-based curriculum consistent with the CEFR. However, there seems little interest on the part of the different national and regional governments in improving on the various school leaving tests currently in use — even though these old fashioned tests provide hardly any information about students’ competences (Amengual Pizarro, 2005, 2006; García Laborda, 2010, 2012; Sanz Sainz, & Fernández Álvarez, 2005), and despite the fact that their replacement is a current requirement of Spanish law. Its future is thus somewhat uncertain. A clear sign of this lack of resolve is that while at present the LOMCE lays out the new proposals for educational reform in the Spanish baccalaureate teaching and assessment, the originally mandatory inclusion of both oral comprehension and production was postponed until 2018 and at present there are no visible indications of such changes to the test. Clearly further discussions are needed if progress is to be made in this area. The lack of a listening component on the present selectividad test is particularly worrying given the importance of oral comprehension both as an essential component of communicative competence and as a contributory factor in successful language acquisition (Rubin, 1994; Vandergrift, 1999; Zhang, 2012). If a final secondary school qualification is to be used for university entrance, this is of obvious importance — especially for those courses which use EMI. Students will be required both to take lecture notes and to take part in question and answer sessions and they therefore need to be adequately prepared for success at these tasks. Due to the importance of the skill of listening for students’ future performance, this thesis will argue that a listening component should be included in the final baccalaureate test for school leavers, and furthermore that such a test should be a valid and reliable CEFR-related measure of students’ proficiency levels. To this end, this thesis both develops a prototype test and provides validity evidence to support the interpretation and use of scores in a school-leaving/university-entrance context. Following a critical review of selected literature on the construct of listening proficiency and listening test tasks, a theoretical model is outlined along with test specifications to produce a test of B2 CEFR- related listening. The proposed test is then evaluated using both quantitative and qualitative methodology within a validity argument framework (Kane, 2012). Results are presented from a Rasch analysis of test scores, which show the test to be a reliable and accurate measure of the unidimensional construct of listening. Verbal report methodology showed that participants used construct relevant knowledge, skills and abilities to answer test items. Expert judges agreed that the test contains B2 level items and a cut score for the test was identified by following the CEFR-linking process recommended by the Council of Europe (CoE, 2009). Opinions canvassed from test takers themselves demonstrated that this group of stakeholders strongly believe that a listening section should be introduced as part of the school leaving/university entrance test. In so doing the present national curriculum as presented in the law would be evaluated. However, it was noted that ongoing research would be necessary to give evidence that such a test results in positive washback on teaching and learning within the Spanish education system.