Principal AI Data & Evaluation Strategist

A leading AI lab is expanding its roster of expert contributors, and Braintrust is helping identify senior experts for this advisory role. We are seeking a highly experienced Principal AI Data & Evaluation Strategist to serve as a consultative expert and thought partner alongside program managers, providing strategic guidance during project discovery, solution design, data planning, evaluation-framework creation, quality-rubric development, and pilot execution.

The ideal candidate brings deep expertise in AI data creation, human evaluation systems, model benchmarking, and quality-measurement methodologies. You will work directly with stakeholders, engineering teams, researchers, and program leadership to define project requirements before execution begins. This is not an operational-delivery role; it is responsible for bringing subject-matter expertise, challenging assumptions, identifying risks early, and ensuring projects are designed for quality, scalability, and measurable outcomes. It is a remote, flexible engagement contributing to frontier AI work.

Responsibilities:

• Strategic advisory: lead project discovery, translate business goals into AI data requirements, define project success metrics, and identify risks and failure modes early.
• Data strategy: design training and evaluation datasets, define taxonomy structures, recommend data-sourcing methodologies, establish ground-truth standards, and identify coverage gaps and edge cases.
• Evaluation design: develop human-evaluation frameworks, create quality-measurement methodologies, establish acceptance criteria, design benchmarking approaches, and build calibration mechanisms.
• Thought leadership: challenge assumptions that may impact project quality, recommend industry best practices, guide teams through ambiguity, and support executive and client reviews.

Requirements:

• 10+ years in AI, data science, machine learning, research, or AI operations.
• Experience designing large-scale AI data programs.
• Experience with LLM evaluation, human feedback systems, AI benchmarking, and dataset development.
• Strong written English and the ability to communicate complex ideas clearly.
• Comfortable working independently in a remote environment.
• Based in India.

Preferred:

• Experience advising on or leading large-scale AI data and evaluation programs.