SACS WorkshopJuly2007

Views:
 
Category: Entertainment
     
 

Presentation Description

No description available.

Comments

Presentation Transcript

Assessment Tools: 

Assessment Tools Jim Eck Associate Vice President for Academic Affairs Rollins College Winter Park, Florida July 24, 2007

Introduction: 

Introduction Who Am I Goals of this workshop (5) Why Assess Assessment Measures SACS Core Requirements and Comprehensive Standards General Education QEP

Why Assess?: 

Why Assess? Public demands for accountability Institutional needs for accreditation National expectations concerning the improvement of higher (especially undergraduate) education

Assessment Essentials: 

Assessment Essentials Agree on goals and objectives for learning Design and implement a thoughtful approach to assessment planning Involve individuals from on and off campus Select or design and implement data collection approaches Examine/share and act on assessment findings Regularly reexamine the assessment process From C. Palomba & T. Banta, 1999

Liberal Education: 

Liberal Education Liberal Education: education for free citizens; promotes the study of history, mathematics, languages, and sciences (American Academy for Liberal Education) Competing and/or Complementary Goals: enhance among students their content knowledge of a specific field of study vs. facilitating thinking and reasoning skills among students so that they can solve ill-structured, real-world problems

What are Student Learning Outcomes?: 

What are Student Learning Outcomes? Encompass a wide range of student attributes and abilities, both cognitive and affective, which are a measure of how their college experiences have supported their development as individuals (R. Frye). http://www.ac.wwu.edu/~dialogue/issue2.html ETS Major Field Tests http://www.ets.org/Media/Tests/MFT/demo/mftdemoindex.html ACT Alumni Outcomes Survey (Self-report) http://www.act.org/ess/fouryear.html ACT College Outcomes Survey (Self-report) http://www.act.org/ess/fouryear.html Collegiate Learning Assessment Project http://www.cae.org/content/pro_collegiate.htm

Major Field Tests: 

Major Field Tests Biology Business Chemistry Computer Science Criminal Justice Economics Education History Literature in English Mathematics Music Physics Political Science Psychology Sociology MBA

ACT Outcomes Surveys: 

ACT Outcomes Surveys Developing problem-solving skills Broaden my awareness of diversity among people, their values and cultures Developing leadership skills Likert Scales Self-report/indirect measure

Collegiate Learning Assessment: 

Collegiate Learning Assessment You are the assistant to Pat Williams, the president of DynaTech, a company that makes precision electronic instruments and navigational equipment. Sally Evans, a member of DynaTech's sales force, recommended that DynaTech buy a small private plane (a SwiftAir 235) that she and other members of the sales force could use to visit customers. Pat was about to approve the purchase when there was an accident involving a SwiftAir 235. You are provided with the following documentation:

Collegiate Learning Assessment: 

Collegiate Learning Assessment 1: Newspaper articles about the accident 2: Federal Accident Report on in-flight breakups in single engine planes 3: Pat's e-mail to you & Sally's e-mail to Pat 4: Charts on SwiftAir's performance characteristics 5: Amateur Pilot article comparing SwiftAir 235 to similar planes 6: Pictures and description of SwiftAir Models 180 and 235

Collegiate Learning Assessment: 

Collegiate Learning Assessment Please prepare a memo that addresses several questions, including what data support or refute the claim that the type of wing on the SwiftAir 235 leads to more in-flight breakups, what other factors might have contributed to the accident and should be taken into account, and your overall recommendation about whether or not DynaTech should purchase the plane.

Assessing Critical Thinking: 

Assessing Critical Thinking ACT: Collegiate Assessment of Academic Proficiency (CAAP): designed to measure gains in critical thinking during the first two years of college http://www.act.org/caap/ California Critical Thinking Skills Test (CCTST): designed to measure critical thinking and reasoning skills; CCTDI http://www.insightassessment.com/test-cctst.html

Assessing Engagement: 

Assessing Engagement National Survey of Student Engagement - http://www.nsse.iub.edu/ Community College Survey of Student Engagement http://www.ccsse.org/ Faculty Survey of Student Engagement http://nsse.iub.edu/fsse/index.cfm College Student Experiences Questionnaire http://www.indiana.edu/~cseq/ College Senior Survey http://www.gseis.ucla.edu/heri/css.html

Assessing the First-Year Experience: 

Assessing the First-Year Experience The First-Year Initiative Assessment College & University Housing http://www.webebi.com/default.aspx Your First College Year http://www.brevard.edu/fyc/yfcy.htm Cooperative Institutional Research Program http://www.gseis.ucla.edu/heri/freshman.html

SACS: 

SACS Core Requirement 2.5: The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that incorporate a systematic review of programs and services that (a) results in continuing improvement, and (b) demonstrates that the institution is effectively accomplishing its mission (Institutional Effectiveness)

SACS: 

SACS Comprehensive Standard 3.3.1: The institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results

SACS: 

SACS Comprehensive Standard 3.4.1: The institution demonstrates that each educational program for which academic credit is awarded (a) is approved by the faculty and the administration, and (b) establishes and evaluates program and learning outcomes

How to Define Desired Student Learning Outcomes: 

How to Define Desired Student Learning Outcomes Top Excellence Academic Achievement Local Community Strong and Distinctive Undergraduate Education Bottom Diverse Spiritual Traditions Environmental Stewardship Responsive Citizenship

How to Define Desired Student Learning Outcomes: 

How to Define Desired Student Learning Outcomes Departmental Goals Students majoring in Theatre Arts & Dance will demonstrate, through jury evaluations and critical reviews, proficiency in Performance/Directing; Dance, Acting, Directing, Choreography, Improvisation (Expressive Arts)

How to Define Desired Student Learning Outcomes: 

How to Define Desired Student Learning Outcomes Departmental Goals Students majoring in Spanish will demonstrate an "advanced" level of aural comprehension skills in accordance with ACTFL standards (Humanities) All students majoring in Education will pass the Florida State Licensure Exam for teachers (Social Sciences)

How to Define Desired Student Learning Outcomes: 

How to Define Desired Student Learning Outcomes Departmental Goals Students, through volunteer work at the Genius Reserve, will demonstrate an understanding of the criteria and process for preserving natural ecosystems (Environmental Studies, Interdisciplinary Studies)

How to Define Desired Student Learning Outcomes: 

How to Define Desired Student Learning Outcomes Students majoring in physics will demonstrate the ability to apply experimental techniques and advanced mathematics to solve problems in both written and graphical form (Sciences and Mathematics)

Assessment Measures: 

Assessment Measures Presentation of directing, choreography, or improvisation projects; Individual Script Analysis Project; Directing or Acting scenes/In class dance presentations; Laboratory application in production; Journals (Theatre) Spanish majors will achieve a rating of "advanced" on a portfolio of writing assignments that will be judged by the Spanish faculty according to ACTFL guidelines

Assessment Measures: 

Assessment Measures 100% demonstration of each Accomplished Practice at the Performance level by the end of the Student Teaching experience (Education) Natural Lands Acquisition Project; students utilize model criteria to analyze the Econlockhatchee River and the Wekiva River. After scoring each site, student scores are contrasted to professional assessments (Environmental Studies)

Assessment Measures: 

Assessment Measures Physics majors in the senior year are expected to perform at or above the 50th percentile on the Major Field Achievement Test (MFT) in physics. 759 others and counting

Use of Assessment: 

Use of Assessment Students have improved learning outcomes related to directing, choreography and performance since the college developed a second stage series of plays that are produced, directed, designed, and acted by students Students have improved their ability to use advanced math to solve problems since the department began requiring students to take Principles of Physics and introductory calculus simultaneously

Assessment at Rollins College: 

Assessment at Rollins College Assessment Matrix See handout http://www.rollins.edu/provost/assessment/ Academic/Administrative Support Rubrics See handout http://www.rollins.edu/provost/assessment/ National Peer Review Guidelines See handout

Consortia Benchmarking: 

Consortia Benchmarking Associated Colleges of the South http://www.colleges.org/ Higher Education Data Sharing Consortium http://www.e-heds.org/ Annapolis Group http://www.collegenews.org/ State Organizations (i.e., Independent Colleges and Universities of Florida, ICUF) http://www.icuf.org/

Other Useful Assessment Resources: 

Other Useful Assessment Resources Southern Association of Colleges and Schools http://www.sacscoc.org/ Association to Advance Collegiate Schools of Business http://www2.aacsb.edu/ National Association of Schools of Music http://nasm.arts-accredit.org/ Association of Theological Schools http://www.ats.edu/

Other Useful Assessment Resources: 

Other Useful Assessment Resources National Council for Accreditation of Teacher Education http://www.ncate.org/ American Academy for Liberal Education http://www.aale.org/ Malcolm Baldrige National Quality Award http://www.quality.nist.gov/Education_Criteria.htm Council for Accreditation of Counseling and Related Educational Programs http://www.cacrep.org/

Other Useful Assessment Resources: 

Other Useful Assessment Resources Assessment Update http://www.josseybass.com/WileyCDA/WileyTitle/productCd-AU.html Indiana University-Purdue University Indianapolis http://www.planning.iupui.edu/conferences/national/nationalconf.html http://www.planning.iupui.edu/conferences/international/internationalconf.html

Other Useful Assessment Resources: 

Other Useful Assessment Resources Alverno (Student Assessment as Learning) http://www.alverno.edu/for_educators/student_as_learn.html North Carolina State http://www2.acs.ncsu.edu/UPA/assmt/ases_ovv.htm Truman State http://assessment.truman.edu/ University of Wisconsin-Stout (MBA) http://www.uwstout.edu/mba/

Other Useful Assessment Resources: 

Other Useful Assessment Resources University of Wisconsin-Madison http://www.provost.wisc.edu/assessment/manual/manual2.html University of Colorado-Boulder http://www.colorado.edu/pba/outcomes/units/unitindx.htm Old Dominion University http://web.odu.edu/webroot/orgs/AO/assessment.nsf/pages/homepage Boise State http://www2.boisestate.edu/iassess/outcomes/arts_sci.htm

Analyzing Assessment Results: 

Analyzing Assessment Results External Measures Psychometric properties (reliability/validity) but often are self-reports that rely on Likert Scales or utilize multiple-choice responses (Bloom’s Taxonomy); indirect/reliable/easy to score Internal Measures Pilot (Construct Validity) Rubrics (Systematic Observation) More difficult to score/information is rich See handout

SACS: 

SACS Comprehensive Standard: 3.5.1: The institution identifies college-level competencies within the general education core and provides evidence that graduates have attained those competencies Structure the curriculum so that no matter what choices a student makes within the general education options, it is not possible to pass all courses and not have demonstrated the competencies (few) that the college has identified.

SACS: 

SACS Core Requirement 2.12: The institution has developed an acceptable Quality Enhancement Plan and demonstrates that the plan is part of an ongoing planning and evaluation process

What is QEP: 

What is QEP The QEP requires a focus on the future An institution should begin work on its QEP at least two years in advance of the scheduled onsite visit The reaffirmation process no longer has an “end”. It is now a continuous process focused on improvement Develop specific, well-defined goals—fewer rather than more. “Dig one well—well and avoid digging a well field Characteristics of QEP: creative, nonlinear, recursive, evolutionary—with the possibility of revisions, changes even as the implementation unfolds; a bit messy, a bit uncertain, exciting, challenging and rewarding (we’re anticipating exciting and rewarding!) More like creating a stone sculpture than assembling a montage Addressing that “one issue” you’ve always ignored about your institution Paradigm shift “in Medias Res”

A Few QEP Topics: 

A Few QEP Topics The Formation of an Academic Resource Center Enhancement of Online Education Partnership in Achieving Student Success (Academic Advising and College Success Skills Course) The Academically Challenging Tutorial Lab Do the Right Thing: A Campus Conversation on Ethics

QEP: Where to Begin: 

QEP: Where to Begin Planning “Rollins College Mission Statement” (pretty) Research Where do the opportunities for improvement lie? What initiative has the highest opportunity cost if you don’t do it? Address a weakness or improve upon a strength

QEP: Assessment of the Plan: 

QEP: Assessment of the Plan The institution demonstrates that it has means for determining the success of its Quality Enhancement Plan The institution identified and used relevant internal and external measures to evaluate the plan, such as changes in learning outcomes and/or comparisons with similar institutions or participation in cooperative plans with other institutions or with projects similar to the QEP The institution identified an internal system for evaluating and monitoring the progress of the plan (QEP Steering Committee) The institution described and incorporated evaluation of the plan The institution described the process that will be used to incorporate evaluation findings into the ongoing enhancement of the institution

QEP: Assessment of the Plan: 

QEP: Assessment of the Plan Translation Develop an assessment schedule that Assigns accountability for assessment activities Establishes an early baseline Incorporates appropriate check points

QEP: Assessment of the Plan: 

QEP: Assessment of the Plan Accountability Provost Assistant Provost (early baseline, check points occur biannually) Deans/Directors QEP Steering Committee By Initiative: Leadership and Citizenship: Director of Community Engagement; Director of Student Involvement and Leadership Retention and Recruitment: Dean of Admission and Enrollment Academic and Social Integrity: Dean of the Faculty Internationalization: Director of International Programs; Director of the International Studies Center Diversity: Director of Multicultural Affairs

QEP: Assessment of the Plan: 

QEP: Assessment of the Plan Establish an Early Baseline Qualitative Measures: (example: first-year/senior essay: What do citizenship and leadership mean to you?) Quantitative Measures: (example: student responses to the Multi-Institutional Study of Leadership)

QEP: Assessment of the Plan Check Points: 

QEP: Assessment of the Plan Check Points

In Medias Res: 

In Medias Res Administrative Changes Occur 2 Presidents, 3 Provosts, Various Trustees Role of QEP Steering Committee Changes Focus on allocating funds rather than assessment Re-constitute QEP Steering Committee, place renewed emphasis on assessment ($1000), seek proposals for funding (proactive rather than reactive) Students graduate and there are other administrative/staffing changes; many of the original “champions” of QEP topic are no longer here

In Medias Res: 

In Medias Res People generate great ideas (theory); basic Often, the people who generate great ideas are not held responsible for implementing those great ideas (action research); applied Similar to creating a mess and then leaving someone else to clean it up; or, here’s the problem, now you design the solution/implementation strategy that makes the problem go away

In Medias Res: 

In Medias Res Difficult to maintain momentum over a couple of years let alone five years No assessment will occur without a long-term plan and without people being held accountable for making sure the plan (as described on paper) comes to fruition QEP evolves and morphs and everyone will try to link their innovations to it (especially if funding is possible)

In Medias Res: 

In Medias Res QEP becomes the vehicle to do “all things” (most institutions can do any one thing they might choose to do but none of us can do everything we’d like to do) Unintended very positive outcomes A much clearer focus, for Rollins, about our institutional purpose Revised Mission Statement (no longer pretty) Guiding Principles

Conclusion: 

Conclusion Assess learning outcomes with multiple measures (qualitative/quantitative; indirect/direct; external/internal) Few general education core competencies QEP Assessment Plan (Accountability, Schedule, Baseline, Check-points) Institutional Integrity 1 year of data may be acceptable Samples of learning outcomes may be acceptable

authorStream Live Help