10+ years of experience in data operations, annotation, or model evaluation
5+ years in management or leadership roles
Proven success scaling data or RLHF operations across
Responsibilities
Build, lead, and scale a global data operations organization including full-time employees, contractors, and vendor partners
Define clear roles, quality standards, and performance metrics across all data functions (evaluation, labeling, RLHF, and generation)
Partner with Legal, Compliance, and Security to ensure all global data work adheres to HIPAA and data privacy standards
Oversee the design and execution of evaluation frameworks for LLMs and agentic behaviors — both automated and human-in-the-loop
Lead data labeling, synthesis, and annotation operations, ensuring medical accuracy, consistency, and context-rich quality
Manage large-scale RLHF pipelines — aligning training data with clinical and ethical objectives
Optimize throughput, cost, and quality across in-house teams and external vendors
Partner with engineering and product to design and improve data operations infrastructure, including labeling tools, quality assurance systems, and task routing platforms
Implement robust QA processes and auditing frameworks to ensure data integrity and reliability
Drive continuous improvement in efficiency, consistency, and evaluator experience
Work closely with Research, Model, and Product teams to define data needs and feedback loops
Collaborate with Clinical and Safety leaders to align annotation and evaluation standards with clinical guidelines
Provide strategic input into data strategy, metrics, and operational planning