A result on its own invites the wrong questions. Is it good? Did it improve because the plan got better, or because the plan changed shape? A comparative report answers some of those questions by placing each program beside its college and the institution, all reviewed against the same standard in the same year.
This note is for program heads reading their first comparative report, and for assessment offices explaining one. Comparative reports are coming next in ProverbAI's first release, so it describes how they are designed to be read. Most of the advice applies to any program review report.
Why reports wait for the whole year
Comparisons need everyone. A college average calculated while half of its programs are still under review would be a different number a week later. So reports are not issued plan by plan. Once every plan that came in has been reviewed and signed off, the reports are produced together, and each program receives one that does not change after it is published. The program head and the assessment office are reading the same document.
One late plan does not hold up everyone else. Once the deadline has passed, the reports can go out without the plans that are not in yet. A plan that arrives later is reviewed in the usual way and gets its own report, compared with the numbers already published and marked as coming after them, so no published report changes.
What a report contains
- The program's overall result and rating for the year.
- Its standing among the institution's programs that year.
- The college and institution averages, for comparison.
- Results for each main part of the plan, such as the curriculum map, the learning outcomes, and the methods and measures.
- Feedback drafted by ProverbAI and approved by a reviewer, with priorities for the next cycle.
How to read it
Five habits make a report more useful than its headline result.
Start with the parts, not the overall result
The overall result tells you where the program stands. The parts tell you why. A part that is missing entirely is often the quickest thing to fix. A part that is present but thin usually needs a conversation with the faculty who collect the evidence.
Compare with the college before the institution
Programs in the same college often share courses, support staff and habits of documentation. If a program trails its college on one part, a neighbouring program has probably solved the problem already. If the whole college trails the institution on the same part, the conversation belongs with the college, not with one program.
Treat standing as relative, not absolute
Standing shows where a program sits among this year's programs. If every program improves, a program's standing can stay the same while its plan gets better. Look at its own results across the years as well: earlier reports sit alongside the new one.
Don't over-read small differences
Ratings are bands. Two programs with almost identical results can fall on either side of a boundary, and that small gap says little about the quality of either plan. Treat the rating as a signal of where attention is needed, and the feedback as the explanation.
Read the feedback with the plan open
Feedback is written against the plan as it was submitted, and each finding cites the passage it rests on. Reading the two side by side is the fastest way to turn a priority into a change: find the section in question, and decide what next year's plan will say.
What a comparison cannot tell you
A program report reviews an assessment plan. It does not measure how much students learned, and a stronger result is not a claim that one program teaches better than another. It says that one plan describes its outcomes, measures and use of results more completely and more clearly than another. That is worth knowing and worth improving, but it is a different thing, and a report should never be read as if it were.
For the assessment office
The full set of reports gives the office a view that no single report shows: where plans across the institution tend to fall short, which colleges are strong in which parts, and which programs would benefit from a conversation before the next cycle. In ProverbAI, that view sits on the Results page, with colleges side by side.
