Applied AI Researcher, Benchmarking at Distyl AI

San Francisco, California, United States

Apply Now

Not SpecifiedCompensation

Senior (5 to 8 years), Expert & Leadership (9+ years)Experience Level

Full TimeJob Type

UnknownVisa

Artificial Intelligence, TechnologyIndustries

Requirements

Experience designing and running evaluations: Built or maintained benchmarks, test suites, or experimental frameworks to measure model or system performance
Statistical and analytical rigor: Design fair, reproducible experiments and extract signal from noisy empirical results
Experience building with models, not just building models: Expertise in compound AI systems, agentic collaboration, and techniques like ensembling, ReAct, graph-of-thoughts, etc
Proven track record of research results: Published in top journals, posted work on Twitter, or similar demonstrable achievements
Uses AI every day: Integrates tools like ChatGPT, Cursor, and Perplexity to accelerate workflow
Strong programming and data analysis skills: Ability to build prototypes and perform experiments to prove effectiveness
Bias towards showing vs telling: Focus on demonstrating AI power today rather than long-term ideas

Responsibilities

Design evaluation frameworks that capture reasoning depth, interaction quality, reliability, and operational impact
Construct benchmarks that reflect real-world complexity, serving as standards for new architectures, techniques, and releases
Explore new paradigms for evaluating intelligent systems: adversarial robustness testing, longitudinal performance tracking, and human-in-the-loop assessment
Investigate how metrics shape model behavior and establish rigorous methodologies for quantifying emergent capability
Drive Distyl’s internal research priorities and contribute to industry-wide standards through insights

Skills

Key technologies and capabilities for this role

AI ResearchBenchmarkingLLM EvaluationEvaluation FrameworksReasoning ModelsMachine LearningPrompt EngineeringReliability Testing

Questions & Answers

Common questions about this position

Is this position remote or hybrid?

The position is hybrid.

What salary or compensation does this role offer?

This information is not specified in the job description.

What key skills are required for the Applied AI Researcher role?

Required skills include experience designing and running evaluations such as benchmarks and test suites, statistical and analytical rigor for reproducible experiments, expertise in building with models like compound AI systems and agentic techniques, and a proven track record of research results.

What is the company culture like at Distyl AI?

Distyl AI features a fast-growing, innovative environment pushing AI utilization in enterprises, with researchers who creatively redefine software use, operate in AI-native ways, and come from strong academic backgrounds with research track records; the company is led by proven leaders from Palantir, Apple, and top labs, partnering with OpenAI.

What makes a strong candidate for this benchmarking researcher position?

Strong candidates have experience designing evaluations and benchmarks, statistical rigor, expertise in building intelligent systems with models like agentic collaboration and techniques such as ReAct, and a proven research track record; they thrive in creative, AI-native environments beyond traditional research.

Distyl AI

Provides customized AI solutions for enterprises

About Distyl AI

Distyl.ai provides artificial intelligence solutions tailored for enterprises, focusing on enhancing productivity and streamlining operations. Their products utilize generative AI and large language models, which can be customized to fit a client's specific data, workflows, and systems. This customization allows for smooth integration with existing technologies. Distyl.ai serves a wide range of clients, including Fortune 500 companies in sectors like Consumer Packaged Goods, Retail, Healthcare, Finance, and Manufacturing, as well as federal agencies. The company’s experienced team, which includes professionals from major tech firms and top AI research institutions, enables them to create scalable solutions that meet high standards. Distyl.ai aims to unlock value for clients quickly, typically within a quarter, by providing tailored AI applications that address unique business needs.

San Francisco, CaliforniaHeadquarters

2022Year Founded

$26.3MTotal Funding

SERIES_ACompany Stage

Government & Public Sector, Enterprise Software, AI & Machine LearningIndustries

11-50Employees

Benefits

Health Insurance

Company Equity

Hybrid Work Options

Professional Development Budget

Risks

Rapid AI advancements by competitors could outpace Distyl AI's integration capabilities.

Reliance on OpenAI partnerships poses risks if partnerships are disrupted.

Complex AI integration may lead to client dissatisfaction if not managed well.

Differentiation

Distyl AI customizes AI solutions to specific client data and workflows.

The company integrates generative AI to enhance human productivity in enterprises.

Distyl AI's team includes experts from Palantir, Apple, and Microsoft.

Upsides

Recent $20M funding boosts growth and demand from Fortune 100 customers.

Strategic partnership with OpenAI enhances AI capabilities for enterprise clients.

Growing interest in AI customization drives market confidence and investment.

Land your dream remote job 3x faster with AI

Try Jobo Free