Applied AI Researcher, Benchmarking at Distyl AI

San Francisco, California, United States

Apply Now

Not SpecifiedCompensation

Senior (5 to 8 years), Expert & Leadership (9+ years)Experience Level

Full TimeJob Type

UnknownVisa

Artificial Intelligence, TechnologyIndustries

Requirements

Experience designing and running evaluations: Built or maintained benchmarks, test suites, or experimental frameworks to measure model or system performance
Statistical and analytical rigor: Design fair, reproducible experiments and extract signal from noisy empirical results
Experience building with models, not just building models: Expertise in compound AI systems, agentic collaboration, and techniques like ensembling, ReAct, graph-of-thoughts, etc
Proven track record of research results: Published in top journals, posted work on Twitter, or similar demonstrable achievements
Uses AI every day: Integrates tools like ChatGPT, Cursor, and Perplexity to accelerate workflow
Strong programming and data analysis skills: Ability to build prototypes and perform experiments to prove effectiveness
Bias towards showing vs telling: Focus on demonstrating AI power today rather than long-term ideas

Responsibilities

Design evaluation frameworks that capture reasoning depth, interaction quality, reliability, and operational impact
Construct benchmarks that reflect real-world complexity, serving as standards for new architectures, techniques, and releases
Explore new paradigms for evaluating intelligent systems: adversarial robustness testing, longitudinal performance tracking, and human-in-the-loop assessment
Investigate how metrics shape model behavior and establish rigorous methodologies for quantifying emergent capability
Drive Distyl’s internal research priorities and contribute to industry-wide standards through insights

Skills

AI Research

Benchmarking

LLM Evaluation

Evaluation Frameworks

Reasoning Models

Machine Learning

Prompt Engineering

Reliability Testing

Distyl AI

Provides customized AI solutions for enterprises

About Distyl AI

Distyl.ai provides artificial intelligence solutions tailored for enterprises, focusing on enhancing productivity and streamlining operations. Their products utilize generative AI and large language models, which can be customized to fit a client's specific data, workflows, and systems. This customization allows for smooth integration with existing technologies. Distyl.ai serves a wide range of clients, including Fortune 500 companies in sectors like Consumer Packaged Goods, Retail, Healthcare, Finance, and Manufacturing, as well as federal agencies. The company’s experienced team, which includes professionals from major tech firms and top AI research institutions, enables them to create scalable solutions that meet high standards. Distyl.ai aims to unlock value for clients quickly, typically within a quarter, by providing tailored AI applications that address unique business needs.

San Francisco, CaliforniaHeadquarters

2022Year Founded

$26.3MTotal Funding

SERIES_ACompany Stage

Government & Public Sector, Enterprise Software, AI & Machine LearningIndustries

11-50Employees

Benefits

Health Insurance

Company Equity

Hybrid Work Options

Professional Development Budget

Risks

Rapid AI advancements by competitors could outpace Distyl AI's integration capabilities.

Reliance on OpenAI partnerships poses risks if partnerships are disrupted.

Complex AI integration may lead to client dissatisfaction if not managed well.

Differentiation

Distyl AI customizes AI solutions to specific client data and workflows.

The company integrates generative AI to enhance human productivity in enterprises.

Distyl AI's team includes experts from Palantir, Apple, and Microsoft.

Upsides

Recent $20M funding boosts growth and demand from Fortune 100 customers.

Strategic partnership with OpenAI enhances AI capabilities for enterprise clients.

Growing interest in AI customization drives market confidence and investment.

Land your dream remote job 3x faster with AI

Try Jobo Free