Primary Trait Analysis and Rubrics

Primary Trait Analysis and Rubrics: Using the Grading Process for Assessment: 

Primary Trait Analysis and Rubrics: Using the Grading Process for Assessment A short guide for faculty, coordinators, program directors, and chairs

Primary Trait Analysis Definition: 

Primary Trait Analysis Definition “Primary Trait Analysis (PTA) is a way to take what we already do—record grades—and translate that process into an assessment device.” B. Walvoord and L.P. McCarthy, 1990.

Primary Trait Analysis Characteristics: 

Primary Trait Analysis Characteristics Utilizes a scoring rubric that can be used to assess any student performance. Is assignment-specific, i.e., the criteria are different for each assignment or test. Is very explicit—allows criteria to be made public and understandable to outside audiences. Can be used inside the classrooms to make criteria and standards clear to both teachers and students.

Key Advantages of PTA: 

Key Advantages of PTA Uses and expands a process that faculty trust and control; Uses information already available; Makes more deliberate and systematic the processes that goes into recording grades; When communicated to students, PTA clearly establishes faculty expectations for learning—and enables very structured feedback to students on their performance; Enables holistic view of performance relative to learning goals of an assignment, a course, or in a program.

Why Expand the Grading Process?: 

Why Expand the Grading Process? Makes explicit the criteria a faculty member uses in evaluating student performance on assignments. Creates richer source of feedback for students. Increases validity, reliability, and linkage to course learning goals. Reduces between-instructor variance in multiple sections of the same course. Enables collection of longitudinal information to answer faculty questions about student learning in their courses and how any new approaches have worked.

Enables Individual and Group-Level Assessment: 

Enables Individual and Group-Level Assessment Components of an assignment are recognized as primary traits to be learned by each student, e.g., experimental design for his project? Primary traits can be assessed at the individual level but also at the group level across all students in a section of course, e.g., how well did students in the course describe the experimental design of their projects?

Enables Demonstration of “Closing the Loop”: 

Enables Demonstration of “Closing the Loop” Because PTA can be applied in situations from the grass-roots course level to the institution level, it is a particularly flexible and powerful assessment device. PTA facilitates demonstration of “closing the loop” by showing how assessment information is being used for ongoing improvements in cases from the course and program level to the entire institution (such as with general education assessment.)

Why Some Faculty Find PTA Worthwhile : 

Why Some Faculty Find PTA Worthwhile Provides data for departmental and general education assessment. Saves time in the grading process Helps diagnose students’ strengths and weaknesses. Helps teaching assistants grade papers consistently. Refines and introduces greater distinction into grading practices.

PTA: How it Works: 

PTA: How it Works To undertake Primary Trait Analysis, individual instructors or groups of faculty construct rubrics representing the level of achievement for each primary trait. Requires development of rubrics for each assignment to be assessed. If used across multiple courses or course sections, a certain level of trust and agreement is necessary among instructors to create rubrics and make this work.

Developing Rubrics for Primary Trait Analysis: 

Developing Rubrics for Primary Trait Analysis Rubric: a matrix-based scoring mechanism Rows: describe component parts (primary traits) of the assignment Columns: describe criteria used to score a student assignment Anchors: Each cell in the rubric contains verbal anchors that describe the level of performance such that different raters will arrive at substantially similar ratings on a particular student work or assignment

Example 1: Rubric Outline for PTA of Open-Ended Problem-Solving: 

Example 1: Rubric Outline for PTA of Open-Ended Problem-Solving

Example 1: Anchors for Problem-Solving Rubric: Identifying the Nature of the Problem: 

Example 1: Anchors for Problem-Solving Rubric: Identifying the Nature of the Problem Excellent: provides information and evidence that may be useful; cites reasons for general agreement or disagreement; cites factors that contributes to uncertainties on available information. Poor: Cites no supporting information; provides no context for understanding nature of problem.

Example 2: Rubric Outline for PTA of Scientific Report Assignment: 

Example 2: Rubric Outline for PTA of Scientific Report Assignment

Example 2: Primary Trait Anchors for Methods and Materials Section of a Scientific Report (Walvoord and Anderson, 1995): 

Example 2: Primary Trait Anchors for Methods and Materials Section of a Scientific Report (Walvoord and Anderson, 1995) 5. Contains appropriate, quantifiable, concisely organized information that allows the experiment to be replicated. All information in the report can be related back to this section. Identifies sources of data. Sequences information appropriately. No wordiness. 4. As above, but contains unnecessary information and/or wordiness. 3. Experiment could be replicated from the information given. All information in the report can be related back to this section. However, fails to identify some data sources and/or has problematic sequencing. 2. Marginally replicable. Parts of basic design must be inferred. Procedures not quantitatively described. Some information in results or conclusions sections cannot be anticipated by reading this section. 1. Describes experiment so poorly it cannot be replicated.

Further Steps in PTA: 

Further Steps in PTA Continue to refine verbal anchors until raters agree on ratings at least 70% of the time. Instructors may use rubrics individually, or several instructors may agree to use the same rubric for similar assignments in various sections of the same course. In very large courses, sampling strategies are appropriate. Once assignments have been scored, look at grouped student scores on each primary trait to determine how well instructional strategies are working.

Further Reading on PTA: 

Further Reading on PTA Milton, Ohmer, Pollio, Howard, and Eison, James. Making sense of College Grades. Jossey Bass, San Francisco. 1986. ISBN 0-87589-687-1. Walvoord, Barbara and Anderson, Virginia. Effective Grading: A Tool for Learning and Assessment. Jossey Bass. San Francisco, 1998. ISBN 0-787-94030-5. Walvoord, Barbara and Anderson, Virginia. An Assessment Riddle. Assessment Update, 7(6), 4-5. Jossey Bass, San Francisco, 1995. Walvoord, Barbara and McCarthy, L.P. Thinking and writing in college. Urbana, IL: National Council of Teachers of English, 1990. Wiggins, Grant. Educative Assessment: Designing Assessments to Inform and Improve Student Performance. Jossey-Bass, San Francisco, 1998. ISBN 0-7879-0848-7. Williams, Jan R. and Polhemus, Craig E., eds., American Accounting Association, “Accounting Education News,” Faculty Development Update, Review of A Tool for Effective Grading: Primary Trait Analysis, Spring 2000.