Singapore AI evaluation meetups: what to ask when reviewers disagree

The problem

Your Singapore AI team receives different quality scores from reviewers, and a generic model demo will not explain whether the disagreement comes from unclear criteria, different context or a difficult judgement.

Make the disagreement visible

Prepare two invented outputs and the criteria that reviewers were asked to apply. Explain which judgement differed: factual support, usefulness, tone or an allowed action. A single average score hides that distinction. Your event question should identify the disagreement rather than ask whether human reviewers or an automated judge are generally better. Give the case an identifier and keep the output version, intended reader, task and available evidence together. Then record each reviewer’s stated judgement and the reason they gave. Separate a reviewer’s observation from your explanation of why they differed. If reasons are absent, ask for them rather than supplying a motive. You are preparing a question that a relevant session can examine, not deciding which person is trustworthy from one number. Keep original customer and internal material outside the public record unless the actual permission and event route cover it.

Use a shared-case comparison card

Make six entries: case and output; task and audience; material each reviewer received; criterion and its meaning; judgement with its stated reason; and unresolved decision. Compare these fields before comparing the scores. An application result judged for usefulness to a new user and the same result judged for exact support from a supplied passage may answer different questions. A missing context document can matter as much as the output. Mark those differences; do not calculate an average that silently makes them one criterion. The card also supports comparing two systems on identified tasks, when the same evaluation basis is actually applicable. It supplies no product benchmark or universal scoring rule.

Ask what information each reviewer receives

Check whether the comparison assumes the same task, source material and expected audience. One reviewer may know a business rule that another never saw. Describe those differences without naming your customers or sharing internal policy. A useful conversation can help you write a clearer review instruction rather than merely choose the reviewer whose score you prefer. In a permitted exercise, record whether both reviewers saw the same source version and task instruction. Do not repair a mismatch by pretending the earlier score used material that arrived later. If you clarify an instruction, retain the original and amended versions with the corresponding assessments. A response can change because the task or available evidence changed; call that a new assessment under the recorded conditions, not automatic agreement with the earlier result. Keep the source of a required business decision separate from a peer’s suggestion about clearer wording.

Expose the criterion difference with a fictional case

All people, passages, outputs and scores here are invented. The practice task asks which collection days a supplied note establishes. The note states Friday collection and says nothing about Saturday. The fictional answer says Friday collection is available and also says Saturday collection is available. Reviewer A marks it acceptable under a rule that checks whether the required Friday information appears. Reviewer B marks it unacceptable under a rule requiring every stated day to be supported by the note. The two reasons can both describe what their stated rule checks; their labels are not assessments under one shared rule. Neither result establishes a fact about a real service. Recording “one accepts, one rejects” leaves the important rule difference hidden.

Write the completed clarification without inventing consensus

Continue the fictional record. Both reviewers receive a clarified practice instruction: assess whether every stated collection day is supported by the supplied note. Both now mark the same answer unacceptable because its Saturday claim has no support in that note. Record that agreement under the new instruction, alongside the two earlier reasons. The clarification settles which criterion these fictional reviewers used in this exercise. It does not establish a universal definition of answer quality, determine whether Saturday is offered outside the note or settle what a real team’s task should require. A separate uncertainty about which source belongs to the actual user task still needs its responsible decision.

Look for a calibration exercise

Choose a Singapore session that describes assessment criteria, review examples or discussion of evaluation limitations. Ask whether participants compare judgements on the same case. A networking reception may introduce experienced people but will not necessarily give them enough time or context to examine why two scores differ. Use a narrow enquiry: “I have a permitted shared case with two stated assessment rules and conflicting judgements. Will this activity let participants compare the task, evidence and rules, and what preparation is required?” Confirm whether the host offers participant-case discussion, only a supplied exercise or general explanation. An advanced evaluator demonstration can be useful background while leaving your criteria unresolved. If two systems are compared, ask whether the same case set, setup, assessment rule and exclusions apply. Use the execution-accounting guide when different attempted or unscored cases make their reports incomparable; that reporting question does not disappear after reviewers agree on a criterion.

Record unresolved decisions honestly

Use the session to identify which criterion needs a clearer example and which question still requires a responsible owner. AI Verify offers a structure for documenting practices and evidence; it does not turn a disputed judgement into a universally correct score. Preserve disagreement where it reveals an unresolved decision instead of reporting workshop participation as validation. Keep three outcome rows: instruction or context clarified; assessments actually made under that version; and decision still outside the exercise. Identify who can decide the real task’s requirements instead of assigning that authority to whoever led the meetup. If a peer proposes another test, record the input, permitted change and observation needed before calling it a result. Check whether an invented replacement preserves the observed failure. Keep the application steps and monitoring not examined in the classroom result visible as separate limits, even when the discussion clarifies one scoring rule.

• AI Verify Foundation: testing framework ↗ · Source checked 20 October 2026

Keep the result attached to the case and next question

Retain the original case, revised instructions if any, reviewer reasons and unresolved field so the discussion can be reconstructed. A wholly invented follow-up note might say, “Same output, clarified support rule; both reviewers reject the unsupported Saturday claim. We did not assess whether the real user task requires information beyond this note.” That is a limited calibration outcome, not a validation badge. If an essential decision remains unanswered, narrow the question or seek an appropriate confirmed review route. Use Boardzip’s free AI event search for candidates, then verify the particular activity and joining conditions. Attendance, a consensus score or a certificate does not itself supply a benchmark, current clinic or approval to deploy.