Voice AI Evaluation Lead

Position Overview

• Location Type: Remote
• Job Type: Full-Time
• Salary: $135K - $165K

Deepgram is the leading voice AI platform for developers building speech-to-text (STT), text-to-speech (TTS) and full speech-to-speech (STS) offerings. Over 200,000 developers build with Deepgram’s voice-native foundational models, accessed through APIs or as self-managed software, due to unmatched accuracy, latency, and pricing. Customers include software companies building voice products, co-sell partners working with large enterprises, and enterprises solving internal voice AI use cases. The company ended 2024 cash-flow positive with 400+ enterprise customers, 3.3x annual usage growth across the past 4 years, over 50,000 years of audio processed, and over 1 trillion words transcribed.

Deepgram is seeking a Voice AI Evaluation Lead to own the benchmarking and evaluation of its voice AI models. This role is crucial for the integrity and impact of Deepgram's AI offerings. You will build robust benchmarking pipelines, produce clear and actionable model cards, and collaborate cross-functionally with research, product, QA, marketing, and data labeling teams to shape how models are measured, released, and improved.

Responsibilities

• Build and maintain scalable benchmarking pipelines for model evaluations across STT, TTS, and voice agent use cases.
• Run regular evaluations of production and pre-release models on curated, real-world datasets.
• Partner with Research, Data, and Engineering teams to develop new evaluation methodologies and integrate them into the development cycle.
• Design, define, and refine evaluation metrics that reflect product experience, quality, and performance goals.
• Author comprehensive model cards and internal reports outlining model strengths, weaknesses, and recommended use cases.
• Work closely with Data Labeling Ops to source, annotate, and prepare evaluation datasets.
• Collaborate with QA Engineers to integrate model tests into CI/CD and release workflows.
• Support Marketing and Product with credible, data-backed comparisons to competitors.
• Track market developments and maintain awareness of competitive benchmarks.
• Support GTM teams with benchmarking best practices for prospects and customers.

Ideal Candidate Profile

You’ll love this role if you:

• Enjoy translating model outputs into human insights that guide product strategy.
• Are motivated by precision, fairness, and transparency in evaluation.
• Have a data-minded approach to experimentation and thrive on uncovering what’s working—and what’s not.
• Take pride in designing clean, repeatable benchmarks that bring clarity to complex systems.
• Get satisfaction from cross-functional collaboration, working with researchers, product teams, and engineers alike.
• Want to shape how we define quality and success in speech AI.
• Are excited by the idea of being a key voice in when—and how—we release new models into the world.

Requirements

It’s important to us that you have:

• Experience designing, executing, and iterating on evaluation pipelines for ML models.
• Proficiency in Python and data analysis libraries.
• Ability to develop automated evaluation systems, whether scripting analysis workflows or integrating with broader ML pipelines.
• Comfort working with large-scale datasets and crafting meaningful performance metrics and visualizations.
• Experience using LLMs or internal tooling to accelerate analysis, QA, or pipeline prototyping.
• Strong communication skills, especially when translating raw data into structured insights, documentation, or dashboards.
• Proven success working cross-functionally with research, engineering, QA, and product teams.

It would be great if you had:

• Prior experience evaluating speech-related models, especially STT or TTS systems.
• Familiarity with model documentation formats (e.g., model cards).

Company Information

Deepgram is the leading voice AI platform for developers building speech-to-text (STT), text-to-speech (TTS) and full speech-to-speech (STS) offerings. 200,000+ developers build with Deepgram’s voice-native foundational models – accessed through APIs or as self-managed software – due to our unmatched accuracy, latency and pricing. Customers include software companies building voice products, co-sell partners working with large enterprises, and enterprises solving internal voice AI use cases. The company ended 2024 cash-flow positive with 400+ enterprise customers, 3.3x annual usage growth across the past 4 years, over 50,000 years of audio processed and over 1 trillion words transcribed. There is no organization in the world that understands voice better than Deepgram.