Training Performance Engineer at OpenAI

San Francisco, California, United States

Apply Now

$250,000 – $460,000Compensation

Senior (5 to 8 years), Expert & Leadership (9+ years)Experience Level

Full TimeJob Type

UnknownVisa

Artificial Intelligence, Machine LearningIndustries

Requirements

Love optimizing performance and digging into systems to understand how every layer interacts
Strong programming skills in Python and C++ (Rust or CUDA a plus)
Experience running distributed training jobs on multi-GPU systems or HPC clusters
Enjoy debugging complex distributed systems and measuring efficiency rigorously
Exposure to frameworks like PyTorch, JAX, or TensorFlow and understanding of how large-scale training loops are built
Comfortable collaborating across teams and translating raw profiling data into practical engineering improvements
Nice to have:
Familiarity with NCCL, MPI, or UCX communication libraries
Experience with large-scale data loading and checkpointing systems
Prior work on training runtime, distributed scheduling, or ML compiler optimization

Responsibilities

Profile end-to-end training runs to identify performance bottlenecks across compute, communication, and storage
Optimize GPU utilization and throughput for large-scale distributed model training
Collaborate with runtime and systems engineers to improve kernel efficiency, scheduling, and collective communication performance
Implement model graph transforms to improve end-to-end throughput
Build tooling to monitor and visualize MFU, throughput, and uptime across clusters
Partner with researchers to ensure new model architectures scale efficiently during pre-training
Contribute to infrastructure decisions that improve reliability and efficiency of large training jobs

Skills

Key technologies and capabilities for this role

Distributed TrainingGPU KernelsPerformance OptimizationCollective CommunicationsI/O OptimizationModel ShardingFault ToleranceCheckpointingOrchestrationObservabilityTensor Data Movement

Questions & Answers

Common questions about this position

What is the salary range for the Training Performance Engineer role?

The salary range is $250K - $460K.

Is this a remote or hybrid position, and where is it located?

This is a hybrid role based in San Francisco, CA, requiring three days in the office per week. OpenAI offers relocation assistance to new employees.

What programming skills are required for this role?

Strong programming skills in Python and C++ are required, with Rust or CUDA as a plus.

What is the team culture like at OpenAI's Training Runtime team?

The team focuses on building a unified, modular runtime to accelerate researchers and enable frontier-scale training, partnering closely with model-stack, research, and platform teams, with success measured by improving both training throughput and researcher throughput.

What qualities make someone thrive in this Training Performance Engineer role?

Candidates thrive if they love optimizing performance, digging into systems to understand layer interactions, have strong Python and C++ skills, and experience with distributed training on multi-GPU systems.

OpenAI

Develops safe and beneficial AI technologies

About OpenAI

OpenAI develops and deploys artificial intelligence technologies aimed at benefiting humanity. The company creates advanced AI models capable of performing various tasks, such as automating processes and enhancing creativity. OpenAI's products, like Sora, allow users to generate videos from text descriptions, showcasing the versatility of its AI applications. Unlike many competitors, OpenAI operates under a capped profit model, which limits the profits it can make and ensures that excess earnings are redistributed to maximize the social benefits of AI. This commitment to safety and ethical considerations is central to its mission of ensuring that artificial general intelligence (AGI) serves all of humanity.

San Francisco, CaliforniaHeadquarters

2015Year Founded

$18,433.2MTotal Funding

LATE_VCCompany Stage

AI & Machine LearningIndustries

1,001-5,000Employees

Benefits

Health insurance

Dental and vision insurance

Flexible spending account for healthcare and dependent care

Mental healthcare service

Fertility treatment coverage

401(k) with generous matching

20-week paid parental leave

Life insurance (complimentary)

AD&D insurance (complimentary)

Short-term/long-term disability insurance (complimentary)

Optional buy-up life insurance

Flexible work hours and unlimited paid time off (we encourage 4+ weeks per year)

Annual learning & development stipend

Regular team happy hours and outings

Daily catered lunch and dinner

Travel to domestic conferences

Risks

Elon Musk's legal battle may pose financial and reputational challenges for OpenAI.

Customizable ChatGPT personas could lead to privacy and ethical concerns.

Competitors like Anthropic raising capital may intensify market competition.

Differentiation

OpenAI's capped profit model prioritizes ethical AI development over unlimited profit.

OpenAI's AI models, like Sora, offer unique video creation from text descriptions.

OpenAI's focus on AGI aims to create AI systems smarter than humans.

Upsides

OpenAI's $6.6 billion funding boosts its AI research and computational capacity.

Customizable ChatGPT personas enhance user engagement and satisfaction.

OpenAI's 'Operator' AI agent could revolutionize workforce automation by 2025.

Land your dream remote job 3x faster with AI

Try Jobo Free