Blind AI response evaluation template.
Use the same prompt and observable success criterion for both responses. Record a decision from each reviewer before revealing which variant produced which response.
Download the editable evaluation project
Define the decision before reviewing
Write a criterion a reviewer can check in the response. For this fictional support case, the customer needs to change their email after losing sign-in access.
Criterion. Point the customer to account recovery and avoid asking them to share a password or one-time code.
The first answer might offer the correct recovery route and explain when to contact support. The second might sound helpful but ask for a code. Under this criterion the recovery route wins. A tie is appropriate if both responses meet the criterion equally well or fail in the same consequential way.
Rotate the order and preserve a reason
Remove identifying text such as model names or version labels from the responses. SageBench alternates order across new cases and reverses it for the next reviewer. A stored decision stays attached to the variant even when it appeared on the right for one reviewer and on the left for another. Write down the line or detail that drove the choice.
Read the votes alongside the case details
A win count shows how often each variant met your criterion better in this sample. Open the individual results to see where reviewers disagreed, then use the CSV to compare their reasons. Review a wider range of real tasks before using the count to make a product change. See OpenAI’s guidance on evaluation design and position bias for more on case selection and reviewer agreement.
SageBench hides labels in the review view. Keep the setup and results views out of a reviewer’s session if you want them to stay blind. Project backups include the variant labels and decisions.