The Gold Standard of Anthropology | Evaluator Selection Panels, Community Nomination and Independence–Expertise Trade-Offs

1. Begin with the evaluation decisions, not a list of famous names

Back to contents

Write down the decisions the evaluation must support, the evidence it will handle and the groups who could be affected. These determine the expertise needed. A health programme may require epidemiology, implementation knowledge, local language and experience of how care is actually accessed. No single résumé is likely to contain the full set.

Turn the brief into a capability matrix before considering candidates. This reduces the risk that prestige, prior relationships or availability quietly become the selection criteria.

2. Separate independence from distance

Back to contents

Independence means freedom to design, conduct and report without improper influence. Geographic or organisational distance can help, but it is not the same thing. An external consultant can depend financially on the commissioner; an internal evaluator can sometimes work under strong disclosure and reporting protections.

Describe structural, functional and behavioural safeguards separately. Ask who controls the contract, methods, data access, publication and response to criticism. A label such as independent should never replace those facts.

3. Build a panel with declared roles and decision rules

Back to contents

A selection panel can include commissioners, evaluation specialists, subject experts and representatives of affected communities. Give each member a defined role, access to the same criteria and a declared route for recording concerns. Decide in advance whether the panel seeks consensus, uses scored criteria or makes a recommendation to a named decision-maker.

Panel diversity is not enough if one member controls the shortlist or undocumented conversations change the result. Retain the longlist, conflict declarations, scores, reasons for exceptions and the final appointment record.

4. Use community nomination without turning one nominee into the community

Back to contents

Community nomination can surface people with language, historical and relational knowledge that a conventional procurement process misses. It can also privilege office-holders, well-connected organisations or people comfortable with formal meetings. Offer several nomination routes and explain what authority nomination does and does not confer.

A nominee may serve as evaluator, adviser, peer reviewer or community liaison. The role should follow capabilities and consent, not a symbolic demand that one person represent every internal difference.

5. Score expertise and safeguards as separate dimensions

Back to contents

Evaluate method competence, domain knowledge, contextual knowledge, communication, accessibility and team leadership. Then assess conflicts, financial dependence, prior programme roles, confidentiality duties and freedom to publish. A high score on expertise should not erase a serious unmanaged conflict; a clean conflict form should not compensate for missing competence.

Use thresholds for non-negotiable requirements and narrative judgement for trade-offs. Publish enough of the matrix for readers to understand why the chosen team could do the work and how its vulnerabilities were handled.

6. Worked example: choosing a team for a youth programme review

Back to contents

A three-person panel shortlists an internationally known evaluator, a local youth worker nominated by participants and a mixed team led by a regional researcher. The famous candidate has strong methods but no language coverage. The youth worker has trust and programme knowledge but supervised part of the service. The mixed team has adequate methods, a bilingual researcher and no delivery role, but needs a paid youth advisory member.

The panel appoints the mixed team, contracts the youth worker as a disclosed contextual adviser rather than sole evaluator, and gives the advisory group access to the draft findings and a written dissent channel. The decision is defensible because the panel preserved both independence and situated knowledge instead of pretending one person maximised every quality.

7. Design recusal and challenge procedures before they are needed

Back to contents

Panel members and candidates should disclose employment, funding, family, organisational, advocacy and prior-study relationships relevant to the evaluation. The process must specify who judges materiality and whether the response is disclosure, restricted access, recusal, added review or exclusion.

A conflict rule that appears only after a contested appointment looks strategic. Apply the same standard to commissioners and community nominees, while recognising that relationship-rich settings may require managed transparency rather than impossible social isolation.

8. Changed-context transfer check: a small rural service network

Back to contents

In a small region, every qualified evaluator may know the service, funder or community leaders. A rule requiring no prior relationship could import someone with little contextual competence and no language access. The transferable standard is not perfect distance; it is an explicit map of relationships plus safeguards proportional to the decisions at stake.

The panel might divide roles, appoint an external methods reviewer, require independent data custody and publish a conflict statement. If no arrangement can protect candid participation or reporting, the limitation belongs in the evaluation claim.

9. Protect less visible voices from nomination capture

Back to contents

Compare nominees with a stakeholder map that includes people who left the programme, declined services, hold minority positions or face access barriers. Provide confidential self-nomination and outreach through more than one intermediary. Compensate participation and make clear that criticism will not affect services.

Do not treat the number of nominations as a popularity vote. Repeated nomination may show trust, but it may also reflect organisational reach. Document who had a realistic chance to nominate and whose absence remains unresolved.

10. Audit the appointment as part of the evidence chain

Back to contents

Retain the capability matrix, invitation text, nomination routes, shortlist, panel membership, disclosures, recusals, scores, reasons, contracts, reporting protections and changes to the team. Check whether the person proposed at selection is the person who actually performs the work.

After the evaluation, review whether missing expertise, access restrictions or relationship pressures affected data collection and interpretation. Selection quality is not proven by a tidy process alone; it should be tested against what happened.

11. Frequently asked questions

Back to contents

Must every evaluator be external? No. Independence depends on authority and safeguards, not only employment status. Does community nomination guarantee legitimacy? No; nomination routes can exclude quieter groups. Can a conflicted expert contribute? Sometimes, in a bounded and disclosed advisory role with independent judgement elsewhere. Should the highest score always win? Only if the scoring model and thresholds genuinely reflect the evaluation task.

12. The gold standard makes the appointment contestable and capable

Back to contents

A strong selection process begins with the evaluation decisions, assembles complementary capabilities, separates expertise from independence, broadens nomination routes and records how trade-offs were resolved. It allows challenge without making one community representative carry every burden.

The aim is not a team with no relationships or perspective. It is a team whose knowledge, obligations, authority and safeguards are visible enough that readers can judge whether the evaluation deserves trust.