Senior Software Engineer, AI Inference Systems at NVIDIA

Toronto, Ontario, Canada

Apply Now

Not SpecifiedCompensation

Senior (5 to 8 years), Expert & Leadership (9+ years)Experience Level

Full TimeJob Type

UnknownVisa

AI, Technology, SemiconductorsIndustries

Requirements

Bachelor’s degree (or equivalent experience) in Computer Science (CS), Computer Engineering (CE) or Software Engineering (SE) with 7+ years of experience; alternatively, Master’s degree in CS/CE/SE with 5+ years of experience; or PhD degree with thesis and top-tier publications in ML Systems, GPU architecture, or high-performance computing
Strong programming skills in Python and C/C++; experience with Go or Rust is a plus; solid CS fundamentals: algorithms & data structures, operating systems, computer architecture, parallel programming, distributed systems, deep learning theories
Knowledgeable and passionate about performance engineering in ML frameworks (e.g., PyTorch) and inference engines (e.g., vLLM and SGLang)
Familiarity with GPU programming and performance: CUDA, memory hierarchy, streams, NCCL; proficiency with profiling/debug tools (e.g., Nsight Systems/Compute)
Experience with containers and orchestration (Docker, Kubernetes, Slurm); familiarity with Linux namespaces and cgroups
Excellent debugging, problem-solving, and communication skills; ability to excel in a fast-paced, multi-functional setting

Responsibilities

Contribute features to vLLM that empower the newest models with the latest NVIDIA GPU hardware features; profile and optimize the inference framework (vLLM) with methods like speculative decoding, data/tensor/expert/pipeline-parallelism, prefill-decode disaggregation
Develop, optimize, and benchmark GPU kernels (hand-tuned and compiler-generated) using techniques such as fusion, autotuning, and memory/layout optimization; build and extend high-level DSLs and compiler infrastructure to boost kernel developer productivity while approaching peak hardware utilization
Define and build inference benchmarking methodologies and tools; contribute both new benchmarks and NVIDIA’s submissions to the industry-leading MLPerf Inference benchmarking suite
Architect the scheduling and orchestration of containerized large-scale inference deployments on GPU clusters across clouds
Conduct and publish original research that pushes the pareto frontier for the field of ML Systems; survey recent publications and find a way to integrate research ideas and prototypes into NVIDIA’s software products

Skills

vLLM

GPU kernels

compilers

CUDA

MLPerf

speculative decoding

parallelism

kernel fusion

autotuning

DSLs

compiler infrastructure

scheduling

orchestration

ML Systems

NVIDIA

Designs GPUs and AI computing solutions

About NVIDIA

NVIDIA designs and manufactures graphics processing units (GPUs) and system on a chip units (SoCs) for various markets, including gaming, professional visualization, data centers, and automotive. Their products include GPUs tailored for gaming and professional use, as well as platforms for artificial intelligence (AI) and high-performance computing (HPC) that cater to developers, data scientists, and IT administrators. NVIDIA generates revenue through the sale of hardware, software solutions, and cloud-based services, such as NVIDIA CloudXR and NGC, which enhance experiences in AI, machine learning, and computer vision. What sets NVIDIA apart from competitors is its strong focus on research and development, allowing it to maintain a leadership position in a competitive market. The company's goal is to drive innovation and provide advanced solutions that meet the needs of a diverse clientele, including gamers, researchers, and enterprises.

Santa Clara, CaliforniaHeadquarters

1993Year Founded

$19.5MTotal Funding

IPOCompany Stage

Automotive & Transportation, Enterprise Software, AI & Machine Learning, GamingIndustries

10,001+Employees

Benefits

Company Equity

401(k) Company Match

Risks

Increased competition from AI startups like xAI could challenge NVIDIA's market position.

Serve Robotics' expansion may divert resources from NVIDIA's core GPU and AI businesses.

Integration of VinBrain may pose challenges and distract from NVIDIA's primary operations.

Differentiation

NVIDIA leads in AI and HPC solutions with cutting-edge GPU technology.

The company excels in diverse markets, including gaming, data centers, and autonomous vehicles.

NVIDIA's cloud services, like CloudXR, offer scalable solutions for AI and machine learning.

Upsides

Acquisition of VinBrain enhances NVIDIA's AI capabilities in the healthcare sector.

Investment in Nebius Group boosts NVIDIA's AI infrastructure and cloud platform offerings.

Serve Robotics' expansion, backed by NVIDIA, highlights growth in autonomous delivery services.

Land your dream remote job 3x faster with AI

Try Jobo Free