Making Self-Studies More Effective

Paradigm shift in accreditation from resource-based indicators of quality to outcomes-based indicators

• Two-tiered regional visits: WASC/SACS
• Data-driven institutional decisionmaking
• Ongoing vs. episodic evaluation
• Emphasis on demonstrating effective student learning

Assessment of student learning is the key principle

Self-study needs to:

• be analytical and evidence-based rather than primarily descriptive .
• include direct as well as indirect measures of student learning.
• Direct: juried portfolios of student work, capstone courses, problem-based learning activities, case study analyses
• Indirect: surveys of student satisfaction with courses, alumni surveys
• Aren't grades enough? The question is: What do the grades measure?

ATS Commission expectation of assessment

It is no longer sufficient to plan to have a plan for assessment. The Commission expects to see the implementation of an assessment plan.

Degree program chapter should reflect the use of direct and indirect measures

• What has been learned?
• What has been successful?
• What is in need of improvement?
• What decisions/strategies are being taken to secure needed improvement?
• Assessment should help the faculty to work smarter, not just more.

Emphasis is on improvement
Perfection and complete resolution of issues are not expected.

Emphasize the analysis of rather than the accumulation of data
Data don't make decisions—people make decisions.

Interpret the findings
Example: If GSQ score indicates that students are dissatisfied with practical skills training, how does the school interpret that finding, and what decision/choices does it make in light of that interpretation?

"Closing the Loop"
"Closing the loop" is a process of gathering data, analyzing the data, making indicated revisions, and revisiting the process. "Continuous Quality Improvement" is a good example.

A good assessment chapter

A good assessment chapter shows how individual course objectives are aligned with degree program goals and how faculty, corporately, take time to think systemically about these alignments and reflect on the results of this corporate thinking. Faculty should examine not only the course implications of individual course evaluations but also the effectiveness of the degree program itself. (Each of the degree program standards has a specific section (the fifth) on evaluation, of which the MDiv standard is illustrative; cf. ATS Degree Program Standard A.5).

• Are there patterns that emerge from the analysis of a random review of portfolios from a particular cohort of students?
• Are there benchmarks for effectiveness (e.g., an expectation that 80 percent of the students will be able to demonstrate ability to apply knowledge of church history to a current, critical incident in the congregation or parish)?
  • What venues or processes are in place to ensure this kind of faculty review of the program?
  • Is there an annual or semi-annual faculty meeting devoted to degree program review?
  • Is the process regular, ongoing, and sustainable?

Be strategic about your assessment activities

It is not necessary to "fix" everything at once. It is more than OK to fine-tune the strategy. The apt Spanish proverb poco a poco means "little by little." Establishing a timeline and targeting specific issues to be addressed are effective ways to calm faculty fears about assessment and its impact on faculty workload.

Good assessment requires multiple measures

Examples:

• direct and indirect measures of learning,
• feedback from denominational leaders about effectiveness of graduates,
• analysis of direct indicators (success rates on denominational ordination/board examinations).

Problems that the accrediting staff members see in their experience with self-studies

• Too much catalog description and not enough analysis of effectiveness
• Demographic data and graduation rates; little direct measures of student learning
• Too much emphasis on institutional resources (faculty degrees, library holdings, and technological capacity), not enough emphasis on outcomes
• Little or no evidence of use of available ATS data resources: SIR, PoM, ESQ/GSQ, IPPR*
• Little evidence that there is in place a process for an ongoing, regular, and sustainable review of the goals of the degree programs; impression that the self-study is a decennial artifact, a portrait frozen in time rather than a "frame" taken from an ongoing reel of evaluative film
• Little attention to how the four themes (evaluation, globalization, diversity, and freedom of inquiry) in the ATS Commission standards inform degree program goals
• Little attention to the "public character" of theological education

* SIR = Strategic Information Report

PoM =  Profiles of Ministry

ESQ = Entering Student Questionnaire

GSQ = Graduating Student Questionnaire

IPPR = Institutional Peer Profile Report