Hickey Assessment KSI

Reflective Assessment/Alignment: 

Reflective Assessment/Alignment Daniel. T. Hickey Learning & Performance Support Lab Department of Educational Psychology University of Georgia AAAS/CCMS Knowledge Sharing Institute June 2003

Two Questions about Assessment: 

Two Questions about Assessment Aligning assessments to standards: How does articulation of outcomes impact assessment? What is the impact of learning performances? Depends on the goal of assessment. Impact of Assessment Practice: How can information be used to revise curriculum? What is the diagnostic power of items? How can assessment advance learning? Depends on the format of the assessment.

CCMS Design Tensions and Curricular Solutions: 

CCMS Design Tensions and Curricular Solutions Learning inquiry vs. content coverage: Focus on coherent “slices” though the systems. Tsou’s analogical comparisons. Self-directed, intrinsically-direct inquiry leads to multiple paths: Focus on select benchmarks. Some problem contexts require non-targeted background knowledge: Make tradeoffs in context selection.

The New Challenge of “Accountability”: 

The New Challenge of “Accountability” NCLB, WWC, likely DOE Technology Plan use standardized M-C tests as policy lever: Bizarre way to demonstrate knowledge. Cheap, reliable, & naïve appeal. Conventional curriculum & integrated teaching systems have a profound advantage: Fine-tuned to directly present many of the associations needed to identify most correct response. Decontextualized, individualized format of turnkey systems lend nicely to random assignment. Innovators run risk of being labeled “unproven”.

Addressing Curricular Tensions with Assessment & Alignment: 

Addressing Curricular Tensions with Assessment & Alignment Good assessment promotes coverage and inquiry. “Learner-oriented” formative feedback: Attaches new content to robust inquiry schema. Promotes participation in discourse and argumentation. Good classroom accountability: Guides student and teacher curricular enactments. Guides refinement of materials and methods. Affords shared knowledge base for subsequent inquiry. Provides additional incentives to motivate engagement. Design-based refinements of activities and classroom assessments ensures needed outcomes.

Functions of Assessment: 

Functions of Assessment Almost all assessment serves formative goals: Ultimate goal is improving teaching & learning. More summative functions get all the attention: Summative functions undermine formative. Range of formative functions: Guiding curricular selection. Guiding curricular revision. Guiding curricular enactment. Directly advancing student understanding.

Reflective Assessment/Alignment Framework: 

Reflective Assessment/Alignment Framework Refined in studies of technology-based innovations: CTGV, GenScope, LBD, VSS, NASA COTF. Aims to increase evidential (summative) and consequential (formative) validity. Considers levels of assessment: Activity, Curriculum, & Standards-Oriented Considers approaches to assessment: Empiricist, Rationalist, Sociocultural.

Stanford’s Five Levels of Summative Testing: 

Stanford’s Five Levels of Summative Testing Immediate: artifacts from the enactment of the curriculum; Close: parallel to the content of a unit; Proximal: Tapping the knowledge and skills of a curricula but not necessarily the topics. Distal. Reflecting state and national standards; Remote: Broader outcomes (e.g., NAEP, workforce)

Levels of Assessment: 

Levels of Assessment Embedded (Immediate ): Is the curriculum. Informal, collaborative, minimally summative Activity-Oriented (Close): Somewhat formal, individually completed. Modestly summative. Curriculum-Oriented (Proximal): Sensitive to specific curriculum and standards. More formal, fairly summative. Standards-Oriented (Distal): Reflects broader standards. Extremely formal and highly summative.

Close Assessment:Activity-Oriented “Quizzes”: 

Close Assessment: Activity-Oriented “Quizzes” Similar context, content, and representation: Screen captures or paper versions of physical lab. Use multiple-choice only when appropriate. Short answer + explanation is a useful format. Ideal for guiding, tuning & revising activities. Completed at the end of a unit (e.g., weekly). Extremely sensitive to enacted activities. Should directly guide enactment, support self-assessment. Ideal for directly furthering student knowledge. Need “learner-oriented” feedback rubrics. Coach participation in “assessment conversation”. Don’t let summative function undermine formative goals.

Proximal Assessment:Curriculum-Oriented “Exams”: 

Proximal Assessment: Curriculum-Oriented “Exams” Cherry picked or custom items and tasks. Same content, but new and formal context. Sensitive to curriculum and standards. Use PALS, NAEP, released forms, text supplement. Ideal for assessing & refining curricular coverage Use to guide refinement of activities and quizzes. Good for classroom accountability (grades). Ideal for comparing enactments of innovation. Can’t give to comparison groups. Useful for furthering student knowledge: Requires students to formalize knowledge. Need summative and formative rubrics.

Distal Assessment:Standards-Oriented “Tests”: 

Distal Assessment: Standards-Oriented “Tests” Randomly-selected items from larger pool: New context and some new content. Reflects broader standards. Same methods used for high-stakes tests. Ideal for summative evaluation and curricular comparisons. Test must be protected from compromise. Different items on pretest and posttest. Provide only score-level feedback.

Approaches to Assessment: 

Approaches to Assessment Assessment practices reflect assumptions about knowledge: Assumptions are often implicit. Competing assumptions = competing tests: Supports validity of high-stakes tests. Explains how innovators compromise validity. Generally 3-4 sets of assumptions: “Behavioral, Cognitive, & Cultural”. “Familiar, Important & Enduring”.

Three Approaches to Assessment: 

Three Approaches to Assessment Behavioral/Empiricist focus on lower-level associations: Recognition and recall of specific associations. Supports multiple-choice standardized tests. Cognitive/Rationalist focus on higher-level conceptual schema. Problem-solving focus on rationalization. Central to 90’s assessment reform Situative/Sociohistoric focus on participation in knowledge practices. Shift from acquisition to participation. Shift from individual to event. Reconcile tensions between two acquisitory approaches

Assessment Levels x Assessment Approaches: 

Assessment Levels x Assessment Approaches

Suggestions for CCMS Alignment: 

Suggestions for CCMS Alignment Clarify functions of assessment: Don’t confound formative and summative. Separate activity-oriented and standards-oriented assessments. Maximize consequential validity of formative. Maximize evidential validity of summative. Discriminate between approaches to assessment. Greeno, Collins, & Resnick; Hickey & Zuiker Wiggins’ familiar, important, & enduring. Exploit incentive value of summative functions. Use IRT (Facets or WinMira) whenever possible.

Suggestions for CCMS Feedback: 

Suggestions for CCMS Feedback Exploit the formative value of assessment: Directly advance student understanding. Guide teacher’s enactment of new activity. Guide students’ enactment of later activity. Develop good classroom assessments: Create learner-oriented rubrics for each assessment (“answer explanations”). Include new content. Coach assessment conversation.

General Suggestions: 

General Suggestions Use design-based methods (Sloane & Gorard, 2003) for iterative refinement: Use quizzes to refine activities; Use exams to refine quizzes and feedback; Use tests to refine curriculum, exams, and exam feedback. Build local-level prototheory along the way. Include a more comprehensive view of assessment in 2061 criteria.

Assessment Levels x Assessment Approaches: 

Assessment Levels x Assessment Approaches

GenScope Project Results: 

GenScope Project Results

GenScope Project Results: 

GenScope Project Results