INTRODUCTION TO THE GUIDELINES
A number of key criteria have been identified as critical elements in test instrument design which influence the overall effectiveness of an ICAO LPR test. Each criterion plays a key role in how well an ICAO LPR test meets the objectives of the LPRs and how well the test performs overall in terms of validity and fairness. All the criteria are important in ICAO LPR best practice assessment.
These Guidelines have been developed to assist civil aviation authorities and organisations involved in the design of LPR tests recognise and understand key issues related to the design of LPR tests and their impact on overall LPR testing practices.
The criteria identified in the Guidelines are core issues that shape the overall effectiveness and suitability of a testing system for the assessment of pilots and air traffic controllers for ICAO LPR licensing purposes.
Since 2003, when ICAO Document 9835 was first published, test developers and civil aviation authorities have relied solely on this Manual for information on how to design and develop or select tests which fulfil the ICAO LPR requirements. The intention was that test developers would work to a common standard based on ICAO Document 9835 while designing, developing and administering tests for licensing purposes. An important objective of the ICAO LPRs is that equivalence between different tests and the ICAO Levels these tests award is achieved.
In an ideal situation, each State would develop/select and implement its own testing system(s) in accordance with the ICAO LPRs to a common standard so that there was confidence of equivalence among States. In other words, all going well, ICAO Document 9835 would provide the basis for a common framework for the implementation of tests, a means to promote equivalence of ICAO Levels awarded in different States and by different tests, and equivalence in the aviation English language skills and language knowledge assessed by these tests. The objective was that a common and universal standard could be implemented globally, irrespective of which tests were used and in which States.
In the field of language testing, the ICAO LPR initiative is unprecedented. There are no other examples of an attempt to develop and implement a common language standard internationally where different tests are developed and implemented independently which are based on a common rating scale and set of associated descriptors. While ICAO Document 9835 provides States, civil aviation authorities and individual test developers with a common framework for the design, development, selection and administration of tests, the Manual itself serves only as guidance material and was not intended to be highly prescriptive. In publishing ICAO Document 9835, ICAO made guidance material available to the aviation industry that enabled the development and implementation of tests to begin, and which would improve proficiency standards. ICAO Document 9835 was developed to facilitate the development of tests that best met local test development capabilities and needs, yet were still based on a common standard. This had never been attempted before on a global scale in any industry.
Looking back at the first ten years of the ICAO LPRs, the project has achieved its goal. States have implemented the ICAO LPRs. Civil aviation authorities, airlines and ANSPs across the globe are now aware of the importance of language proficiency in safety and have systems to assess personnel. Language proficiency is now recognised in aviation operations and language training is being implemented in response to the ICAO initiative. Standards are improving. To this effect, ICAO Document 9835 has achieved its objective.
However, based on this history and reflections on current practices, all stakeholders in the ICAO LPRs should work together to look at aspects of implementation which require strengthening and/or improvement. One of the most noticeable needs is a means to foster greater standardisation and harmonisation of testing standards.
Since the early 2000s, a multitude of ICAO LPR tests have been developed and implemented in isolation. And, while they may have been developed in response to the ICAO LPRs and followed the guidance material outlined in ICAO Document 9835, it is to be expected that differences in interpretation of the guidance material, testing methods and indeed what constitutes an effective LPR test instrument would emerge. As a result, differences in testing standards and practices have arisen. This has introduced a number of challenges which could undermine the long-term effectiveness of the ICAO LPRs and aviation safety.
These challenges include:
- inadequate implementation of a uniform ICAO LPR standard internationally,
- variations in what language skills and language knowledge ICAO LPR tests assess and the extent to which these relate to the language, communicative contexts and proficiency levels needed for safe aeronautical communications,
- a lack of equivalence between the ICAO Levels issued by different tests,
- market forces favouring the emergence and spread of inferior tests and testing practices at the expense of quality tests and testing practices,
- a lack of confidence or even mistrust in the ICAO Levels awarded by different tests and/or in different States,
- a proliferation of tests based on flawed or unsuitable test designs which negatively impact on language training programmes and long term attitudes towards language proficiency and the ICAO LPRs,
- threats to civil aviation safety.
This lack of consistency among LPR tests stems from wide differences in test instrument design. Further, regulatory authorities do not yet have access to a common, detailed framework to evaluate LPR tests, that could act as a tool towards the closer standardisation of LPR testing systems.
The key to establishing a good quality testing system starts with the design of the test instrument. In order for equivalence between ICAO LPR tests to be established to facilitate harmonisation of LPR standards internationally, ICAO LPR tests need to incorporate a number of key test instrument design elements. By identifying and highlighting these key elements, based on ICAO Document 9835 and best practice in language assessment, the Guidelines aim to provide clear parameters for tests, so that they have more in common in terms of the language skills and language knowledge they assess, and are more closely aligned with language assessment needs for air-ground aeronautical communications. In simple terms, these Guidelines aim to define baseline test instrument design elements that need to be included so that worthwhile comparisons can be made between tests. This can then facilitate international and inter-test standard setting.
Only when an ICAO LPR test instrument is well designed can there be the possibility for the development and implementation of a sound and valid testing system. Often it is incorrectly assumed that it is the quality of the raters which determines the quality and effectiveness of a testing system. In fact, rating can only be effective if the test instrument is well designed, valid and effective, because this is what informs the type, range and complexity of language, coverage of skills and contexts of communication for assessment purposes.
These Guidelines aim to provide clarification and explanation of key issues related to test instrument design in order to limit any potential confusion in the interpretation of ICAO Document 9835. They provide in-depth explanations of why key criteria are critical to the effectiveness of a test. The Guidelines may expand on, go further than or narrow the scope of statements made in ICAO Document 9835. They aim to reduce opportunities for markedly different interpretations of ICAO Document 9835 that impact on ICAO LPR test design and result in variations between testing systems. These Guidelines are developed in response to issues that have emerged since 2003, that have caused perceived divergence in LPR testing practices.
The Guidelines are framed around, but not exclusively based on what is contained in ICAO Document 9835. They aim to provide a common, detailed framework for CAAs to critique, vet and select LPR tests, and for test developers to design and develop tests in line with the ICAO LPRs and best practice in language testing. They aim to provide a clear explanation of best practice issues in test instrument design and promote greater consistency and more uniform application of the ICAO LPRs standards, all of which is reflected at the core of ICAO LPR testing systems – in the design of the test instrument.
The ICAEA Board has been involved with and followed the history of the ICAO LPRs since their inception. Over the last 15 years, ICAEA’s efforts have focused extensively on providing guidance to initially educate the industry then more recently improve LPR testing quality.
These Guidelines have been carefully researched and developed by members of the ICAEA Board who have many years of extensive academic and practical experience in both mainstream and ICAO LPR language testing. The ICAEA Board is committed to upholding the aims of the ICAO LPRs and best practice in language testing, while also taking account of the practical issues involved in the development and implementation of ICAO LPR tests in this ongoing emerging field of language testing. This background and thinking has shaped how the Guidelines have been developed.
A language test instrument is the tool/device which is administered during a language assessment to collect information about a person’s language skills and abilities to allow these to be measured. A test instrument is made up of sets of test tasks and test items constructed and assembled in a meaningful way so that the language level(s), language knowledge and language skills the test is designed to measure occur in a predetermined and controlled way. To people administering or taking tests, test instruments are typically presented as a list of scripted questions asked by an interlocutor, questions to be answered on paper or a computer, or any combination of these.
The design of the test instrument is outlined in a test specifications document – the blueprint for the design of the test instrument and the development of all the test versions in the test bank. A test developer needs to develop test specifications as an important part of a test development project. The design of the test instrument affects all aspects of the quality of a language testing system and is a fundamental requirement to best practice in language testing. The test instrument is the cornerstone of quality in a testing system and determines the validity and fairness of a test.
In fact, it is not possible for a test to achieve high validity if the test instrument is poorly designed.
Because the design of an LPR test is a prerequisite to all aspects of LPR testing – including delivery, administration and rating/scoring, these Guidelines focus on issues related to test instrument design.
For each criterion the following is provided:
|1.||Key Issues & Considerations
An overview of the criterion and an explanation of why it is important.
|2.||ICAO Statements & Remarks
A reference to what may have previously been mentioned in ICAO Document 9835 on issues directly and indirectly related to the criterion, if applicable.
|3.||Why this issue is important
An explanation of the importance of the issue and how it influences the quality of a test.
|4.||Best Practice Options
An explanation of options for best practice when addressing how the criterion needs to be considered in the design of a test.
A list of key references from mainstream language testing literature that provide additional explanations for why the criterion is important.
ICAEA will publish the Test Design Guidelines on the ICAEA website in early 2019.
Using the ICAO LPR Test Design Guidelines
ICAEA is scheduling a series of practical, regional Workshops in 2019 and 2020, to introduce and demonstrate the use of the Guidelines to stakeholders.
Each 2.5-day workshop will feature a programme especially designed for personnel involved in LPR testing, from licensing authorities and test developers.
CRITERIA FOR TEST INSTRUMENT DESIGN