When a grants round produces inconsistent scores, a panel that cannot reach consensus, or a final list of recommended grants that does not clearly reflect the stated criteria, the natural response is to look at the people involved. Were the assessors well briefed? Did the convenor manage the discussion effectively? Did a particular panel member dominate?
Sometimes that diagnosis is correct. More often, the problem is upstream. The rubric was not specific enough to produce comparable scores across assessors. The panel structure conflated individual scoring with group deliberation. The criteria were weighted verbally during panel discussion rather than structurally during the design of the round. And the conflict of interest process was a declaration form rather than an enforceable mechanism.
These are process design problems. The people are doing their best with the tools they have been given. Fixing the people problem — better briefing, a more assertive convenor — produces marginal improvement. Fixing the process design problem produces a reliable, repeatable, defensible assessment process.
Failure 1: Inconsistent scores across assessors. Application A receives a score of 8 from one assessor and a 4 from another, on the same criterion. In the panel review, the discussion goes in circles because the two assessors are using different mental benchmarks, not because they disagree on the merits.
The cause is almost always rubric design, not assessor disagreement. A criterion that says "quality of methodology" gives assessors nothing to anchor their scoring to. What does a 5 look like vs a 7 vs a 9? Without anchor descriptions, assessors construct their own frameworks. Those frameworks differ. The scores are not comparable.
Failure 2: Panel deliberation dominated by individual advocacy. A panel member who has read an application carefully and formed a strong view advocates for it during discussion. Other panel members, less certain, defer. The outcome reflects the most vocal assessor rather than the collective assessment.
The cause is usually a panel structure that does not separate independent scoring from collective deliberation. When assessors score in front of each other — or score as part of the same session in which they discuss — social dynamics influence individual scores. The fix is structural: independent scoring first, deliberation second, and a clear protocol for what deliberation is meant to resolve.
Failure 3: The final recommended list does not map back to the criteria. The scores suggest one set of priorities. The panel's recommended list reflects another. When a governance board asks how the final list was arrived at, the answer involves a lot of "the panel agreed that..." rather than "the criteria weighted X at 40% and the top-scoring applications on that criterion were..."
The cause is usually a breakdown between quantitative scoring and qualitative deliberation — either the criteria were not specific enough to produce meaningful scores, or the deliberation phase was permitted to override the scores without any structured protocol for how or when overrides are appropriate.
A scoring rubric that produces comparable results across assessors has three components: criteria, weight, and anchor descriptions.
Criteria are the dimensions on which applications will be assessed. They should map directly to the objectives of the funding programme. If your programme is designed to support community-led environmental initiatives, your criteria might be: community rootedness, environmental impact, organisational capability, and value for money. If your programme supports health research, your criteria might be: scientific merit, investigator track record, methodology robustness, and significance of the research question.
The number of criteria matters. Too many criteria produce fatigue and noise. Four to six substantive criteria is typically sufficient for most contestable funding rounds.
Weight reflects the relative importance of each criterion to the funding objectives. Not all criteria are equally important. If your programme is explicitly designed to support innovative approaches, then innovation should carry more weight than administrative compliance. If team capability is the primary predictor of successful delivery in your context, it should carry more weight than the project design.
Weights should be set before the round opens, published to applicants, and applied consistently across all applications. A round where "Innovation 40%, Methodology 30%, Team 30%" are the stated weights — and those weights are applied structurally in scoring, not just referenced conceptually — produces scores that are directly comparable and that reflect the stated priorities of the programme.
Anchor descriptions are the most commonly omitted component and the most important for inter-assessor consistency. For each criterion at each score level — or at least at key levels (low, mid, high) — write a brief description of what an application at that level looks like.
For example, for a "Community Impact" criterion:
- Score 1-3: The projected community impact is vague or unmeasured; the application does not clearly identify who will benefit or how outcomes will be assessed.
- Score 4-6: The application identifies a clear community need and describes expected outcomes, but the measurement approach is underdeveloped or the target community is not well defined.
- Score 7-9: The application identifies a well-evidenced community need, describes specific and measurable outcomes, and proposes a credible method for assessing impact.
With anchor descriptions, two assessors reading the same application will land within one or two points of each other, not five or six. The discussion in panel becomes about genuine interpretive differences, not about different scales.
Equal weighting is simpler but produces rankings that do not reflect your programme's actual priorities. If innovation is what you are funding, and you score five criteria equally, a mediocre application with a strong compliance score can rank above a genuinely innovative application with a weaker administrative section.
Use weighted criteria when: your programme has a clear primary objective that is more important than secondary considerations; you want the scoring process to produce results that reflect those priorities without requiring deliberation to correct for them; and you are prepared to publish and defend the weights.
Use equal weighting when: the criteria are genuinely of equal importance; or the programme is so early-stage that you are still discovering what predicts good outcomes and do not want to bake in assumptions prematurely.
In most cases, weighted criteria produce better results. The discipline of deciding the weights forces the clarity of purpose that many programmes lack at the design stage.
Scoring and deliberation are different cognitive tasks and should be separated structurally.
Scoring is individual and analytical. Each assessor reads each application and assigns scores against criteria. This should happen independently — assessors should not see other assessors' scores or discuss applications before their own scoring is complete. The purpose of independent scoring is to get an uninfluenced view from each assessor.
Deliberation is collective and interpretive. Once scores are in, the panel reviews the aggregate results, identifies applications where there is significant scoring variance, and discusses those applications specifically. The purpose of deliberation is not to re-do the scoring but to understand and resolve genuine interpretive disagreements.
In a well-structured process, the panel convenor presents the aggregate scores before discussion begins. The discussion is focused on applications with high variance, borderline cases, and any applications where the panel has specific knowledge that should inform interpretation. The deliberation produces a recommendation; the scores produce the evidence for that recommendation.
External assessors — subject-matter experts who review applications outside the panel — should have their own portal that shows only the applications assigned to them. They should not see other assessors' scores, should not have access to the broader application pool, and should not be part of the deliberation phase unless invited in specifically. Their role is to provide expert scoring, not to participate in the funding decision.
A conflict of interest process should produce two things: a structural record of each assessor's declared interests in relation to the current application pool, and an enforcement mechanism that prevents a conflicted assessor from accessing the applications they have a conflict with.
A COI declaration form — whether on paper or in a spreadsheet — produces only the first of these. It records that an assessor declared a conflict. It does not prevent them from receiving the application in their assessment pack, reading it before the conflict was caught, or scoring it before someone noticed the declaration.
An enforcement mechanism means: when an assessor declares a conflict with an applicant, the system automatically removes that application from their assessment queue. They cannot view the application, submit a score on it, or access the panel summary for that application. The conflict declaration, the restriction, and the identity of who made the decision are all recorded in the system.
This is not just a risk management measure; it is a fairness measure. Applicants should be able to trust that no assessor who had a prior relationship with them — professional, personal, or financial — had any influence over the assessment of their application. When the COI process is structural rather than administrative, you can demonstrate that to any applicant, any board member, or any auditor who asks.
Blind review — anonymising applicant names and institutional affiliations during the assessment phase — is a complementary mechanism. It does not replace COI management but reduces the likelihood of bias in applications where a relationship might not rise to the level of a formal conflict but could still influence scoring.
The end of an assessment round should produce a record that answers the following questions clearly:
If you cannot answer all of these questions from your assessment record, your process has gaps. Those gaps may not matter in a routine round. They will matter if a decision is challenged, if you are subject to audit, or if a staff member who holds the institutional memory of the process leaves.
The organisations that manage this well — including health research funders such as the Neurological Foundation and Cure Kids — treat the assessment record as a product of the round, not a byproduct. The record is designed, structured, and maintained as carefully as the funding round itself.
A purpose-built grants management system supports this by design: scores are captured against structured criteria, COI declarations are linked to specific applications, the panel summary is generated from the data rather than constructed after the fact, and the entire record is accessible for the life of the fund.
If you are redesigning your assessment process and want to see how a structured workflow supports it in practice, book a demo.