Diagnostic Testing and Test Construction

Diagnostic testingand testconstruction: 

Diagnostic testing and test construction

Slide2: 

A diagnostic test (sometimes called a formative or progress test) checks on students progress in learning particular elements of the course. (p.6) The content of a diagnostic test is quite specific , referring back to recent classwork. It is intended to have positive results for the student by encouraging him with success … (p. 49) Harrison, A.(1983) A Language Testing Handbook, Macmillan

Slide3: 

Diagnostic tests are used to identify students’ strengths and weaknesses. They are intended primarily to ascertain what further teaching is necessary. At the level of broad language skills this is reasonably straightforward . We can be fairly confident of our ability to create tests that will tell us that a student is particularly weak in, say speaking as opposed to reading in a language. Indeed existing proficiency tests may often prove adequate for the purpose. Hughes, A.,(1989:12), Testing for language teachers, CUP

Slide4: 

We may be able to go further, analysing samples of a student’s performance in writing or speaking in order to create profiles of the student’s ability with respect to such categories as “grammatical accuracy” or “linguistic appropriacy”. Hughes (op.cit. p13)

Slide5: 

But it is not so easy to obtain a detailed analysis of a student’s command of grammatical structures, something which would tell us, for example, whether she or he had mastered the present perfect/past tense distinction in English. Hughes (op.cit. p.13)

Slide6: 

Diagnostic tests seek to identify those areas in which a student needs further help. These tests can be fairly general and show, for example, whether a student needs particular help with one of the four language skills; or they can be more specific, seeking perhaps to identify weaknesses in a student’s use of grammar. These more specific diagnostic tests are not easy to design since it is difficult to diagnose precisely strengths and weaknesses in the complexities of language ability. For this reason there are very few purely diagnostic tests. Alderson, J.C., Clapham,C., & Wall, D., Language Test Construction and Evaluation, CUP

What are the problems?: 

What are the problems? What does this question tell us? I ………… here since January. am living b. live c. was living d. lived e. have lived If the students get it wrong, that they don’t recognise the use of the present perfect for events extending from the past to the present So far, so good.

And if they get it right ?: 

And if they get it right ? Nothing – it could be by chance. But presuming it isn’t : that they recognise that use of that form in that context.

Slide9: 

Whether they would use the form spontaneously for that use in productive activities. Whether they would hear the form in connected speech when a contraction is used. Whether they would use the contraction themselves. Whether they would use the third person singular correctly. Whether they would form the negative correctly. Whether they would form the interrogative correctly. Whether they would form short form answers and questions correctly. Whether they understand the for/since distinction Whether they would recognise/use the present perfect for other uses – eg past event with present result. What doesn’t it tell us?

Slide10: 

The size of a (comprehensive diagnostic test) would make it impractical to administer in a routine fashion. For this reason, very few tests are constructed for purely diagnostic purposes and those that there are do not provide very detailed information. Hughes (op.cit. p.14)

Slide11: 

Diagnostic tests are closely related to progress tests, but their emphasis is on a closer study of specific aspects of performance to establish the causes behind learning progress – or lack of it - so that remedial action can be taken… Diagnosis of spoken text problems may involve probing phonic features such as the perception and interpretation of phonemes, or stress and intonation in connected speech... In diagnostic testing, then, we can draw extensively on the prior specification of language skills; the more detailed and accurate the specification, the more effective should be the diagnosis… Carroll, (1980) B.J., Testing Communicative Performance, Pergamon Press

So ??: 

So ?? A diagnostic test, however inadequate, can help us set objectives We are not operating in a vacuum but may already have considerable knowledge of our students and the course objectives when the test is administered We are planning a course with a set number of hours – we can’t do everything anyway The test therefore does not need to survey every grammatical structure / lexical item and subskill that exists, but can be narrowed down considerably

How ?: 

How ? Needs Analysis + knowledge of Ss’ general level Setting of possible objectives for the course Deficiency analysis of language items/subskills related to those objectives only in order to select priority objectives

The test specification: 

The test specification 1. Statement of the problem Ss are doctors (oncologists) and known to be at upper intermediate level. They have asked for a twenty hour course to improve their ability to understand presentations by native speakers at international medical conferences. A diagnostic test is to be carried out as part of the deficiency analysis which will determine the priority objectives for the course. Face validity is important as the doctors were dissatisfied with a previous programme and have emphasised that they do not want “another language course”. A maximum of 50 minutes can be devoted to the test.

The Test Specification: 

The Test Specification Definition of the content of the test. May include : - language items to be tested - strategies and subskills to be tested - text types to be used - topics to be used - addressees Definition of the format and timing See also Hughes (op.cit) Ch.7

The Test Specification: 

The Test Specification In preparing the diagnostic test for the doctors you would need to analyse the subskills needed to understand this type of presentation, eg : - distinguishing between main points and asides - identifying markers of progression between topics - identifying markers of addition, concession, cause, effect, exemplication etc - understanding phonological features of connected speech such as catenation, elision etc - recognising cohesive devices – eg pronoun reference, synonymy etc - understanding features of spoken English such as ellipsis and so on

The test specification: 

The test specification Types of text /addressees / topic : one or two 10-20 minute authentic medical conference presentations on oncology, by native speakers and addressed to an international specialist audience (eg Youtube – http://www.youtube.com/watch?v=JXt5Ga2WqZI )

The test specification: 

The test specification Format and timing (50 mins) 30 items, focusing on the subskills identified as typically needed to understand the genre. The recording will be divided into chunks with three test items allocated to each chunk. Each chunk will be played twice. Testees will have the chance to Read the test items before hearing the relevant chunk Listen/watch the presentation Listen again while they complete the items

The test specification: 

The test specification Criterial levels of performance : ??? Scoring procedures – objective; a key will be prepared. Inaccuracies not related to listening (eg spelling) will be ignored.

Designing the test: 

Designing the test Sampling Choice of text(s) : two ten minute presentations rather than one longer one would be preferable to diminish the possible effect of “speaker idiosyncracy” Coverage of subskills

Designing the test: 

Designing the test Content coverage One item in each group of three will test gist comprehension. The other two will test detailed comprehension, focusing on the subskills identified in the Test Content section

Testing the test: 

Testing the test Moderation and pre-testing The test will be moderated by a team of two other teachers who will look for issues affecting validity, reliability and practicality - eg : clarity of rubric and test items alternative responses ability to arrive at the correct answer through eg knowledge of the world, visual clues rather than using the subskill supposedly being tested feasibility of time allowed The test will be administered to : a native speaker doctor an advanced level non-native speaker doctor currently following a 121 course

Summary: 

Summary What is diagnostic testing? The difficulties of diagnostic testing The place of diagnostic testing in the course design process Writing the test specification Testing the test

Slide24: 

This presentation is part of the Delta Module One on-line course run by Business Talk Milan. If you would like to find out more about our on-line and face-to-face training courses for the Cambridge ESOL Delta Modules, please contact us via our website : www.business-talk.it