Michelli

Quality Counts:Assessment and Renewal of the Education of EducatorsWisconsin Educational Preparation: Quality Counts ConferenceNovember 2, 2006: 

Quality Counts: Assessment and Renewal of the Education of Educators Wisconsin Educational Preparation: Quality Counts Conference November 2, 2006 Nick Michelli Presidential Professor The Graduate Center The City University of New York

The questions I will raise:: 

The questions I will raise: Why should we engage in quality assurance of the education of educators? What is the place of large scale assessment? Are there examples? What have we learned? What are the promises and pitfalls? How do we get started?

New York City and Wisconsin???: 

New York City and Wisconsin??? Focus on personal experiences in assessing education of educators and policy issues 20 years in large state university with urban, rural and suburban constituency. Urban and Rural—the commonalities: Isolation and limited access to cultural agencies Inadequate resources Difficulty in attracting the best teachers Quality of medical care Transportation access

Why should we engage in the assessment of the education of educators?: 

Why should we engage in the assessment of the education of educators? Education is the cornerstone of democracy. We know that the quality of the teacher is the single most important factor in student learning. We have a public and moral obligation to be certain our programs are as good as they can possibly be.

Why should we engage in the assessment of the education of educators?: 

Why should we engage in the assessment of the education of educators? Assessment can help create a shared vision and be the vehicle for the renewal of programs for the education of educators and of education.

Why should we engage in the assessment of the education of educators?: 

Why should we engage in the assessment of the education of educators? Policy makers are increasingly interested in outcomes data. If we don’t engage in real assessment, someone else will do it for us, and it will not be good!!

A Perfect Storm : 

A Perfect Storm NCL B Pressure for high stakes testing at all levels Pressure for “scientifically based research” The Levine Report: Educating School Teachers (www.edschools.org) USDOE Commission on Higher Education Report: A Test of Leadership Pressure on Accreditors: The dispositions controversy

The Levine Report: Educating School Teachers: 

The Levine Report: Educating School Teachers Focus on student achievement as the primary measure of teacher education program success. …generate a data base that can be used to assess and improve the performance of education schools by providing information on the performance of the teachers and principals who were prepared at the institution….[and] assess which types of teacher education are most effective

The Levine Report: Educating School Teachers: 

The Levine Report: Educating School Teachers Close failing teacher education programs, strengthen promising programs and expand excellent programs by creating incentives for outstanding students and career changers to enter teacher education at doctoral universities.

USDOE Commission on Higher Education Report: A Test of Leadership: 

USDOE Commission on Higher Education Report: A Test of Leadership Student achievement, which is inextricably connected to institutional success, must be measured by institutions on a “value-added” basis that takes into account students’ academic baseline when assessing results.

USDOE Commission on Higher Education Report: A Test of Leadership: 

USDOE Commission on Higher Education Report: A Test of Leadership The results of student learning assessments, including value-added measurements,…should be made available to students and reported in the aggregate publicly.

USDOE Commission on Higher Education Report: A Test of Leadership: 

USDOE Commission on Higher Education Report: A Test of Leadership Higher education institutions should make aggregate summary results of all postsecondary learning measures, e.g., test scores, certification and licensure attainment, time to degree, graduation rates, and other relevant measures, publicly available in a consumer-friendly form as a condition of accreditation.

USDOE Commission on Higher Education Report: A Test of Leadership: 

USDOE Commission on Higher Education Report: A Test of Leadership The collection of data from public institutions allowing meaningful interstate comparison of student learning should be encouraged and implemented in all states.

Why large scale assessments?: 

Why large scale assessments? There is economy of scale. They allow aggregating enough data points to make assessment meaningful. They are more likely to gain access to the complex data needed. They provide a basis for meaningful cross-institutional and cross-pathway conversations. They can provide grounding for research for program renewal. BUT, there are serious limitations

Two Alberts: 

Two Alberts “Not everything that can be counted counts, and not everything that counts can be counted.” --Albert Einstein “What matters is what we measure.” —Albert Shanker That is NOT the same as “We measure what matters.”

Mary Diez’s Three UmpiresA Baseball/Assessment Parable: 

Mary Diez’s Three Umpires A Baseball/Assessment Parable Three Umpires’ Perspectives I call them as they are! I call them as I see them! They aren’t until I call them!!

Slide17: 

The Pathways Study Primary Researchers: Boyd, Grossman, Lankford, Loeb, Wyckoff Study Initiated by: Michelli and Asher, CUNY OAA (RFP $600,000 seed money, Initial questions) Balance Funded by: CUNY, NYSDOE, NSF, Spencer, Carnegie ($5 Million) www.teacherpolicyresearch.org

Why this study?: 

Why this study? The shortage of teachers in urban (and often rural) schools is extraordinary. We must be certain that all pathways they are high quality. In NYC, the majority of new mathematics and special education teachers are through the New York City Teaching Fellows program. I wanted to demonstrate that programs with broader goals do as well as other programs on student achievement measures.

Some Questions: 

Some Questions Who enters which pathway and why? Who enters which teaching job and why? Which features of teacher preparation programs are most effective in improving mathematics and science performance of elementary and middle school children? Who stays in teaching and for who long? Who leaves teaching and why? Who leaves urban teaching, why, and where do they go? What are the relative costs of different pathways (cost effectiveness)?

Slide21: 

Participants and Stakeholders New York City Department of Education New York State Education Department 17 schools of Education (100 programs) Bank Street; CUNY: Brooklyn, City College, Hunter, Lehman, Medgar Evers, Queens, Staten Island, York; Fordham; Hofstra; Long Island; Mercy; NYU, Pace; St. Johns; Teachers College 3 alternative certification programs Teaching Fellows, TFA, TOP United Federation of Teachers

Slide22: 

Study Design for the Project One Labor Market NYC; Focus on Cohort fall 2004 Administrative data All NYC teachers 1990-2006 Student achievement 2000 on; value added math & ELA grades 4-8. Teacher Surveys Selective program participants (2004), all new NYC teachers (2005), second year teachers (2006) Topics: Background and Prior Experiences, Selection of Pathway, Preparation, Professional Development/Mentoring, School Context, Job Selection, Knowledge and Beliefs, Teaching Practices, Retention Program analysis – doc, surveys, interviews State documents, program documents, accreditation reports, interviews, surveys. 17 institutions that prepare most traditional route teachers for NYC; over 120 variables that define programs Alternate route programs that prepare about 25 percent of NYC teachers

So Far: 

So Far College recommended teachers outperform others in the first year in mathematics (exception is math grades 6,7,9 where fellows do better). College recommended teachers outperform others in first year in ELA at all grade levels studied. Differences flatten between CR and TF by 3rd year, when TF have completed teacher education programs. Growth in effectiveness by this measure virtually stops after the third year.

NEXT QUESTIONS: 

NEXT QUESTIONS What makes the difference between pathways? Within pathways? Extensive qualitative studies underway, including doctoral dissertations. What can we say about cost effectiveness?

Slide25: 

Grades 4-8 Teacher Attrition Following 1st Four Years of Teaching, 1999-2003 Cohorts Adjusted for grade School and Year

Slide26: 

School, Grade, and Year Fixed Effects (97&95% match with CR vs. 62&46%) Aright=b0+b1Aisgj(t-1)+Kitb2+Cjg2+Ejg1+Pjg1+πg+πt+πs+εisgjt

Slide29: 

What have I learned? What are the promises and pitfalls? How do we get started?

Recommendations: 

Recommendations If we are to succeed in the education of educators, we need faculty in education, faculty in arts and science, and faculty in P-12 schools with a shared vision working together for positive outcomes. The same principle holds true for assessment. Actions must be supported by policy and structures. Patterson, Michelli and Pacheco. Centers of Pedagogy: New Structures for Educational Renewal. Jossey-Bass, 1999.

Recommendations: 

Recommendations Faculty must be involved in every step of the process, understand the benefits, and be in a position to claim ownership. Where they exist, unions must be involved and informed.

Recommendations: 

Recommendations Collectively, participants at a particular setting must be clear about their conception of the public purposes of education. This is a critical starting point. Academic content knowledge? Critical thinking? Enculturation into a socially just democracy? Preparation leading to nondiscrimination and nonrepression? Preparation to lead rich and rewarding personal lives? Michelli and Keiser, Teacher Education for Democracy and Social Justice. Routledge, 2005. The National Network for Educational Renewal

Recommendations: 

Recommendations The state and local school districts, as well as higher education institutions must be committed to assessment if the kinds of data needed are to be available.

Recommendations: 

Recommendations Recognize that assessment is always scary! Colleges and school systems, especially individual faculty, must feel safe in quality assurance efforts. The focus should be on program improvement, not the assessment of individuals. We must guard against comparisons by agreeing up front how the outcomes can be used.

Recommendations: 

Recommendations Make everything as public as possible within agreements: goals, instruments, interpretations of outcomes, use of data. This is very complex work that requires a variety of assessment methods and data. Use an independent panel to review conclusions before they are released. www.teacherpolicyresearch.org

Recommendations: 

Recommendations Chances are many pathways to teaching will be required to meet projected shortages. One goal should be to find out what works best from each pathway, so all are better. We need to end the holy war among pathways.

Recommendations: 

Recommendations It is likely that you will use value-added measures, but these are very complex, work with large data bases, require a good deal of interpretation, are expensive, and take a long time to develop. Be careful that value-added data doesn’t overwhelm other data.

Recommendations: 

Recommendations If at all possible, those carrying out any large scale assessment should have no vested interests in the outcomes.

Recommendations: 

Recommendations Determining cost effectiveness of pathways is a useful and important public policy goal.

Recommendations: 

Recommendations Address the hard to measure but important outcomes clearly and explain your thinking.

Providing Evidence of Hard to Measure Outcomes: 

Providing Evidence of Hard to Measure Outcomes The Lincoln Center Institute: School of the Arts and Imagination Dr. Scott Noppe-Brandon Dr. Madeline Holzer Dr. Maxine Greene What are the outcomes of aesthetic education? Aesthetic Capacities Deep Noticing: identifying and articulating layers of detail in a work of art through continuous action with it over time. Embodying: to experience a work of art through your senses, as well as emotionally, and also to physically represent that experience

Providing Evidence of Hard to Measure Outcomes: 

Providing Evidence of Hard to Measure Outcomes Questioning: to ask questions throughout your explorations that further your own learning; to ask the question, “What if?” Identifying Patterns: to find relationships among the details you notice, group them, and recognize patterns Creating Meaning: to create your own interpretation based on the previous capacities.

Providing Evidence of Hard to Measure Outcomes: 

Providing Evidence of Hard to Measure Outcomes Teaching for Social Justice Marilyn Cochran Smith and associates at Boston College are developing measures to look at social justice.

Providing Evidence of Hard to Measure Outcomes: 

Providing Evidence of Hard to Measure Outcomes Social and emotional education is being recognized as having a significant connection to both violence reduction and enhanced academic learning. Can it be measured??

Providing Evidence of Hard to Measure Outcomes: 

Providing Evidence of Hard to Measure Outcomes The Center for Social and Emotional Education (CSEE) has developed a reliable and valid instrument for measuring school climate as evidence of social and emotional health involving students, faculty, parents, and administrators. There may be some application of these sorts of instruments, especially in the assessment of school administrator preparation. In our national policy scan done in conjunction with The Education Commission of the States, Wisconsin shows as one of two states with a comprehensive definition of school climate.

Providing Evidence of Hard to Measure Outcomes: 

Providing Evidence of Hard to Measure Outcomes The Comprehensive School Climate Inventory. The Center for Social and Emotional Education. Dr. Jonathan Cohen, President www.csee.net/climate

What does teaching for democracy mean?: 

What does teaching for democracy mean? Learning to listen carefully to other positions. Learning to empathize with others. Learning to think critically about important issues. Learning to argue well for one’s positions. Learning to modify positions with new information.

What does teaching for social justice?: 

What does teaching for social justice? What does Father Coughlin have to do with this? Social justice as nondiscrimination. Social justice as nonrepression. Social justice as opening up life’s chances to all students.

Recommendations: 

Recommendations No large scale assessment can account for the needs of all participants. Individual colleges and universities and their P-12 partner must undertake assessment that is local, comprehensive, reflects their vision on the public purposes of education and leads to the simultaneous renewal of education and the education of educators.

Recommendations: 

Recommendations Reconsider faculty reward systems to assure that research on effectiveness counts in the reappointment, tenure, and promotion processes.

How do we start?: 

How do we start? Develop a clear shared vision of the public purposes of education and focus on student learning, broadly conceptualized. Define clearly what you mean by ideas such as academic learning, democracy, social justice, critical thinking, aesthetic education, and social and emotional education. Think about what evidence of success would look like.

How do we start?: 

How do we start? Study the existing research and use it as a starting point. Cochran Smith and Zeichner, Studying Teacher Education, AERA and Erlbaum, 2005. Darling-Hammond and Bransford, Preparing Teachers for a Changing World, Jossey Bass, 2005.

How do we start?: 

How do we start? Don’t try to reinvent the wheel. Look at and learn from Ohio California Louisiana New York City The Teachers for a New Era Sites

Final Recommendation: 

Final Recommendation Take advantage of this once in a career opportunity to shape how Wisconsin will assess the education of educators to improve learning for all students.

Let me know how it goes!nicholas.michelli@gc.cuny.edu917-882-7670: 

Let me know how it goes! nicholas.michelli@gc.cuny.edu 917-882-7670