Study Domain Expert — Judge Calibration

Role Overview

Mercor is running a structured judge-calibration study to measure how frontier AI models perform on realistic professional work and how experienced practitioners judge that work. We are sourcing experts across Human Resources, Legal, Operations, IT, Administration, Marketing, Sales, Accounting, and Finance.

What You'll Do

• Sign an NDA before receiving any materials.
• Complete a realistic, domain-specific task using supplied source files.
• Review model-generated outputs for the task, rank and score them, and provide evidence-based rationales.
• Provide structured feedback on the task and study design.

Time Commitment 13 hours during the study week, September 14–18, 2026.

Study Conditions

• No rubric or golden answer is provided; professional judgment is the measurement.
• Use of LLMs or other AI assistants is prohibited at every stage.

Who We're Looking For

• 5+ years of hands-on professional experience in one of the listed domains.
• Currently or recently practicing, with clear, evidence-grounded written judgment.
• Able to complete the study within the stated work window.

Eligibility Restriction People who have worked on Project Alchemy in any capacity, or had access to its source-world data, are not eligible for this study.