Requirements

Candidates must have significant experience training large models using Python and PyTorch, including practical experience with the full development pipeline from data processing to training and inference. They should have experience profiling GPU and CPU code in PyTorch for optimal device utilization, writing and improving highly parallel and distributed PyTorch code for large generative models, and working with transformer models and attention implementations. Familiarity with high-performance Triton/CUDA and writing custom PyTorch kernels is preferred, along with experience in high-performance parallel C++ and building inference or demo prototype code.