Pat DeVito NASBE Presentation

Slide 1: 

Thinking About Comprehensive Assessment: Research Syntheses from the National Research Council Pat DeVito Measured Progress June 13,2009 NASBE Study Group

Goals for the presentation : 

Goals for the presentation Explain the NRC Study Process Summarize the main concepts of the Knowing What Students Know and related reports Discuss challenges in building comprehensive assessment systems

National Research Council Study Process : 

National Research Council Study Process Committee Selection Study Process Report Review Process Report Production

Validity and Related Considerations : 

Validity and Related Considerations Measurement Validity – whether a test is valid for a particular purpose, and whether it accurately measures the test taker’s knowledge in the content area being tested. High Stakes: Testing for Tracking, Promotion, and Graduation, National Research Council, 1999.

Slide 5: 

The important thing about a test is not its validity in general, but its validity when used for a specific purpose. Tests are not perfect. No single test score can be considered a definitive measure of a student’s knowledge. Neither a test score or any other kind of information can justify a bad decision.

How People Learn report : 

How People Learn report Summarizes major changes in conceptions about learning Examines the implications of these changes for designing effective teaching and learning environments.

Committee Chairs : 

Committee Chairs Co-Chairs Jim Pellegrino, Vanderbilt/UIC Bob Glaser, University of Pittsburgh (Credit and thanks to Jim for supplying materials on the KWSK work!)

The Committee’s Objective : 

The Committee’s Objective To help establish a theoretical foundation for the design and use of new kinds of assessments that will help all students learn and succeed in school. “Needed are assessments that make as clear as possible to students, their teachers, and other education stakeholders the nature of their accomplishments and the progress of their learning.”

About the Report : 

About the Report Proposes a vision of educational assessment based on contemporary understandings of how people learn and how to measure such learning. Describes an improved approach to assessment design and use, along with promising examples that include applications of technology. Provides directions for research, development, policy, and practice for moving the field of assessment forward.

Assessment as a Process of Reasoning from Evidence : 

Assessment as a Process of Reasoning from Evidence cognition model of how students represent knowledge & develop competence in the domain observations tasks or situations that allow one to observe students’ performance interpretation method for making sense of the data observation interpretation cognition Must be coordinated!

Why Synchrony Is Needed : 

Why Synchrony Is Needed Problem 1: Sophisticated statistical techniques used with restricted models of learning or restricted cognitive tasks will produce limited information about student competence. Problem 2: Assessments based on a complex and detailed understanding of how students learn will not yield all the information they otherwise might if the statistical tools available to interpret the data, or the data themselves, are not sufficient for the task.

Scientific Foundationsof Educational Assessment : 

Scientific Foundationsof Educational Assessment Advances in the Sciences of Thinking and Learning -- the cognition vertex informs us about what observations are important and sensible to make Contributions of Measurement and Statistical Modeling -- the interpretation vertex Informs us about how to make sense of the observations we have made

Slide 15: 

What Cognitive Science Has to Offer Ways of Articulating Multiple Explanations of Thought & Behavior Behavior ranges from micro-processes of rapid perception to macro-processes like problem solving and negotiation Time periods over which behavior and learning unfolds can vary tremendously Multiple Levels of Explanation Cognitive Accounts of Individual Processes and Knowledge Representations Situated/Sociocultural Accounts of Collective Processes and Distributed Knowledge Representations

Metacognition : 

Metacognition Competent performers consciously keep track of their own thinking and adjust their understanding while they learn—a process called metacognition. Self-aware learners can explain which strategies they used and why less competent students monitor their thinking sporadically and ineffectively. Implication - the assessment of metacognitive skills and strategies should be an important assessment target

Multiple Paths to Competence : 

Multiple Paths to Competence Not all children learn in the same way or follow the same paths to competence. children’s problem solving strategies become more effective over time and with practice the growth process is not a simple, uniform progression, nor is there movement directly from erroneous to optimal solution strategies. Implication - assessments should identify specific strategies children are using and where they fall on a continuum of efficiency and appropriateness for a given domain.

Generalizations About Performance : 

Generalizations About Performance Competence & Expertise performance develops in communities that value certain forms of knowledge and activity, like modeling in science. knowledge is tuned to specific patterns of activity, like solving certain kinds of problems. performance increases in scope and precision with multiple, contextualized experiences. no magic levers: practice, disciplined inquiry. Implication -- Assessments must be designed to capture the complexity of competent performance, ranging from mental processes to participation in forms of practice

Generalizations about Knowledge : 

Generalizations about Knowledge Disciplinary Knowledge Is organized in ensembles that facilitate its use. Is amplified by processes of self regulation, or “metacognition,” where learners spontaneously evaluate their knowledge and its limits. Is developed in communities that foster identity and interest. Implication -- Assessments need to reflect multiple “questions” about forms and states of knowing. The questions range from knowledge of specific facts, procedures, and schemas to issues about awareness, reflection and evaluation of personal states of knowing, and how, when, and why we engage in specific practices.

Generalizations about Development : 

Generalizations about Development Not all children learn in the same way or follow the same paths to competence. Conceptual change is often not a simple, uniform progression, nor is there movement directly from erroneous to optimal solution strategies. Intermediate forms of knowledge may not resemble expert forms, so simple building block relations may not hold. Participation often “starts at the edges” and becomes progressively more aligned with core disciplinary practices. Implication - Assessments should identify specific strategies and forms of activity with respect to the role they play in developmental trajectories.

Preconceptions & Mental Models : 

Preconceptions & Mental Models Students come to the classroom with knowledge representations containing pre-conceptions about how the world works. If their initial understanding in a domain is not engaged they may fail to grasp new concepts & information that are the focus for learning. Implication - assessment practices must be designed that can draw out student mental models so teachers can work with their students’ preexisting understandings

Lionni’s Fish is Fish : 

Lionni’s Fish is Fish

The Fish’s Image of Birds : 

The Fish’s Image of Birds

The Fish’s Image of Cows : 

The Fish’s Image of Cows

The Fish’s Image of People : 

The Fish’s Image of People

Some Analogs to Fish is Fish : 

Some Analogs to Fish is Fish Young children who believe the earth is flat…. Physics students who assume “force of the hand” when a ball is thrown into the air Student beliefs that history is about the “good guys” vs the “bad guys” Students at multiple ages beliefs about seasons -- distance from sun not tilt

Why Cognitive Science Models of Knowledge are Critical : 

Why Cognitive Science Models of Knowledge are Critical Tell us what are the important aspects of knowledge that we should be assessing. Give deeper meaning and specificity to standards Give us strong clues as to how such knowledge can be assessed Suggest what can and should be assessed at points proximal or distal to instruction Can lead to assessments that yield more instructionally useful information within and across levels and contexts of use Can guide the development of systems of assessments Comprehensive, Coherent & Continuous

Assessment Design & Use : 

Assessment Design & Use Three “big issues” that remain in translating the scientific foundations into “engineering design” realities: Principles to guide the process of assessment design and development role of the student model Multiple uses of assessment practices must be connected to contexts and purposes Feasibility possibilities for applying technology to critical design and implementation challenges

Opportunities Affordedby Technology : 

Opportunities Affordedby Technology Computer and telecommunications technologies provide powerful new tools necessary to meet many of the design and implementation challenges implied by the merger of current cognitive models and measurement methods: going beyond conventional practices for item presentation implementing a range of task designs and item formats tapping a broader repertoire of cognitive skills and knowledge recording and scoring complex aspects of behavior

Key features of a new approach to assessment design : 

Key features of a new approach to assessment design A model of cognition and learning is a cornerstone of the assessment development enterprise. The design and selection of tasks should be guided jointly by cognition and interpretation elements. The process of construct validation should rest, in part, on evidence that tasks actually tap the cognitive content intended.

Slide 31: 

Assessments should be designed from the beginning to ensure that desired types of information will be possible and effective. Testing must be fair, i.e. one that yields comparable valid inferences from person to person and group to group.

Some Closing Thoughts! : 

Some Closing Thoughts! Better balance and coordination between large-scale & classroom assessment practices Instructional & assessment materials that incorporate knowledge about domain specific learning trajectories Not using what we currently know Teacher education -- preservice & inservice Specifics of quality assessment practices & how they work General pedagogical shift regarding the role of assessment Technology supports Make the assessment process doable and manageable Educating policymakers -- limits and uses of tests

Slide 33: 

We have come a long way in developing assessments but there is much more work to be done to connect the pieces (e.g. classroom, large scale, formative), to lessen overall testing burden and increasing the effectiveness of instruction.