language testing - standardized testing

STANDARDIZED TESTING : 

Beh Hue Ming Dharshini Vasudevan Muhammad Azhan Zaffuan Nur Diana Rofeik Kamini S. Selvaindran STANDARDIZED TESTING

WHAT IS STANDARIZATION? : 

A standardized test presupposes certain objectives or criteria, that are held constant across one form of the test to another. A good standardized test is the product of a thorough process of empirical research and development. It is typical of a norm referenced test. WHAT IS STANDARIZATION?

Slide 3: 

Schools in the United States College entrance exams [SAT] Graduate school exam [GRE] TOEFL IELTS ~They specify a set of competencies for a given domain & constructing validation.

Slide 4: 

Many standardized tests conform to a multiple-choice format. -determining correct and incorrect responses -for large-scale tests Equally involved in certain human-scored tests of oral production and writing. Test of Spoken English [TSE] Test of Written English [TWE]

ADVANTAGES : 

Ready made Frees the teacher from having to spend hours creating a test. Administration to large groups can be accomplished within reasonable time limits. - Scoring procedures are streamlined [for multiple-choice formats] ADVANTAGES

DISADVANTAGES : 

Inappropriate use of such tests -using an overall proficiency test as an achievement test simply because of the convenience of the standardization. Potential misunderstanding of the difference between direct and indirect testing. Some tests include tasks that do not directly specify performance in the target objective. ~ TOEFL DISADVANTAGES

DEVELOPING A STANDARDIZED TEST : 

Determine the purpose & objectives of the test Design test specifications Design, select & arrange test tasks/items Make appropriate evaluations of different kinds of items Specify scoring procedures & reporting formats Perform ongoing construct validation studies DEVELOPING A STANDARDIZED TEST

Slide 8: 

DETERMINE THE PURPOSE & OBJECTIVES OF THE TEST

Slide 9: 

Expected to provide high practicality in administration time money

Examples… : 

TOEFL – to evaluate the English proficiency of people whose native language is not English MUET – to measure the English language proficiency of pre-university students ESLPT – to make placements, to provide diagnostic information about students UPSR – Year 6 students going on Form 1 GET – to enter graduate-level courses − objectives are specific TEYL Examination – able to choose students who are really qualified Examples…

Slide 11: 

DESIGN TEST SPECIFICATIONS

Slide 12: 

Decisions need to be made on how to go about structuring specifications of the test Research must be carried out – may occupy weeks, months or even years

Examples… : 

MUET – emphasis on pronunciation (speaking), spelling (writing), grammar (writing) & listening UPSR – Multiple-choice questions, essay, reading comprehension TEYL Examination – essay writing & interview (speaking) Examples…

Slide 14: 

DESIGN, SELECT & ARRANGE TEST TASKS/ITEMS

Design, Select, Arrange Test Items : 

After specifications  design, select, arrange items. Specs = blueprint – determine the number, types of items to be created. Example K: TOEFL Each item be coded for content and statistical characteristics. Ensure examinee receives questions that assess a variety of skills Cover a variety of subject matter Design, Select, Arrange Test Items

Slide 16: 

A team then designs, selects, adapt items from a bank of items (deposited by free-lance writers and ETS staff) Items are designed to test overall comprehension, certain specific information and inference. Refer to sample, page 74-75 Items are then piloted and scientifically selected to meet difficulty specifications Also, to meet s desired discrimination index

Slide 17: 

Example B: ESLPT 2 subsections to elicit writing performance Summary and response to reading Need to: select appropriate passages, provide appropriate prompts, process data from pilot testing To conform to standards of content validity, difficulty

Slide 18: 

Example C: GET Prompts are designed by a faculty of committee of examiners (specialists) Assumption: topics are universally appealing, capable of yielding intended product of an essay No pilot testing of prompts is conducted Conditions for administration remain constant See sample prompt, page 76-77

Make appropriate evaluations of different kinds of items : 

Practicality, reliability, facility = prominent. Practicality – clarity of directions, timing of test, ease of administration, time required to score responses Reliability – employment of (more than one) scorer, time span for evaluation Facility – key to validity. Unclear directions, complex language, fuzzy data  higher level of difficulty Make appropriate evaluations of different kinds of items

Slide 20: 

Case A: TOEFL The IF, ID, efficiency statistics of the multiple-choice items = not publicly available information The essay portion undergoes scrutiny for its practicality, reliability, facility.

Slide 21: 

Case B: ESLPT Some evaluative impressions of the effectiveness of prompts and passages are gained from informal student and scorer feedback Multiple-choice editing passage showed the value of statistical findings determining the usefulness of items Pointing administrators toward revisions

Slide 22: 

Case C: GET Test of written ability with a single prompt Observe the practicality and facility of questions No data collected from students on their perceptions But scorers can reflect on the validity of a given topic

Specify scoring procedures and reporting formats : 

Specify scoring procedures and reporting formats

Slide 24: 

(A) TOEFL tests - the most straightforward scoring procedure - most complex issues of validation, design and assembly - scores are calculated and reported for a) 3 sections of the TOEFL - the essay rating is combine with the structure and written expression score b) a total score - range 40 to 300 on the computer-based TOEFL and 310 to 677 on the paper- pencil TOEFL. c) essay score is provided on the examinee’s score record.

Slide 25: 

ESLPT - reports a score for each of the essay sections with different rating scale. - each essay is read by 2 readers - editing section is machine-scanned and scored with a total score and with part scores for each of the grammatical/rhetorical sections. - from these data: =placement administrators have adequate information to make placements =teachers receive some diagnostic information on each student in their classes. - students do not receive their essays back.

Slide 26: 

GET - read by 2 trained readers - scores between 1 and 4 according to the scales. 4. Superior 3.competent 2.weak 1. inadequate ----------------------------------------------------------- - the 2 readers scores are added to yield a total score of 2 to 8. - 6 above is allowed to pursue graduate-level courses below: repeat/remedial classes - students receive neither their essays nor any feedback other than the final score.

Perform ongoing construct validation studies : 

Tests are produced in equated forms forms must be reliable across tests such that a score on a subsequent form of a test has the same validity and interpretability as its original. Perform ongoing construct validation studies

Slide 28: 

(A) TOEFL program - cooperate with other tests produced by ETS - over the years dozens of TOEFL - sponsored research studies have appeared in the TOEFL Monograph Series - TOEFL from a Communicative Viewpoint on Language Proficiency = examined the content characteristics of the TOEFL from a communicative perspective based on current research in applied linguistics and language proficiency assessment

Slide 29: 

(B) ESLPT - placing students reliably by means of an essay, multiple- choice grammar, vocabulary test - the security of the latter became suspect, the faculty of the administrators wished to see some content validity achieved in the process. - 2000, the process began with a group of graduate students (Imao et al., 2000) in consultation with faculty members and continued to fruition in the form of new ESLPT. - the process involved a lengthy process of both content and construct validation = face practical issues as scoring the written sections and a machine-scorable multiple choice answer sheet

Slide 30: 

GET - for construct validation, its administrators rely on a stockpile of research on university- level academic writing test (TWE) - in recent years some criticism of the GET has come from international test-takers (Hasoya, 2001) who posit that the topics and the limits of the GET, among other factors, work to the disadvantage of writers whose native language is not English. - this validity issues remain to be fully addressed in a comprehensive research study.

Slide 31: 

STANDARDIZED LANGUAGE PROFICIENCY TESTING

Slide 32: 

“Tests of language proficiency presuppose a comprehensive definition of the specific competencies that comprise the overall language ability”

Language Proficiency Testing… : 

Operational definition of ability Offers a multidimensional view by referring to 3 linguistic traits: - Grammar - Discourse - Sociolinguistics Takes a holistic and more unitary view of proficiency in describing 4 levels: - Superior - Advanced - Intermediate - Novice TOEFL ACTFL Language Proficiency Testing…

Slide 34: 

ACTFL

Slide 35: 

Four Standardized Language Proficiency Tests Test of English as Foreign Language (TOEFL) Test of English for International Communication (TOEIC) International English Language Testing System (IELTS) Michigan English Language Assessment Battery (MELAB)

MUET : 

MUET Developed in 1999 in response to requests for a standardized test of English language proficiency for students applying to enter into university Malaysian University English Test (MUET) is conducted by the Malaysian Examinations Council, a statutory body established under the Malaysian Council Act 1989. To measure a candidate’s productive and receptive skills and the knowledge of grammar and vocabulary in the context of language use

BIBLIOGRAPHY : 

Malaysian University English Test: Issues and Concerns (By: Zuraidah Mohd Dom University of Malaya) Brown H.D. (2004) Language Assessment: Principles and Classroom Practices. Pearson Education: NY. BIBLIOGRAPHY

Slide 39: 

~ THANK YOU ~