Senior Computational AI Architect

A leading AI lab is expanding its roster of expert contributors, and Braintrust is helping identify senior experts for this advisory role. The Senior Computational AI Architect serves as a strategic advisor for language, reasoning, and evaluation-focused AI programs, responsible for designing and validating annotation guidelines, evaluation rubrics, quality frameworks, and reviewer instructions so that AI data and evaluations are accurate, consistent, and scalable.

Working alongside program managers, AI strategists, and stakeholders, you will provide early-stage guidance on task design, quality standards, multilingual considerations, and evaluation methodologies, helping shape project requirements and quality expectations before execution begins. This is a remote, flexible engagement contributing to frontier AI work.

Responsibilities:

• Instruction design: develop annotation guidelines, create reviewer handbooks, design evaluator training programs, and establish escalation frameworks.
• Rubric development: design evaluation rubrics, build reasoning-assessment frameworks, create multilingual quality standards, and define quality acceptance criteria.
• Linguistic consulting: review project instructions, identify ambiguity risks, improve linguistic consistency, and guide localization efforts.
• Evaluation oversight: assess evaluator agreement, review pilot outputs, identify systematic quality issues, and recommend rubric improvements.

Requirements:

• Master's or PhD in Computational Linguistics, NLP, Linguistics, or Cognitive Science.
• 8+ years of experience.
• Expertise across NLP, LLM evaluation, human annotation systems, localization, multilingual AI, and quality frameworks.
• Strong written English and the ability to communicate complex ideas clearly.
• Comfortable working independently in a remote environment.
• Based in the United States.