Principal Software Engineer - Inference as a Service at NVIDIA

Santa Clara, California, United States

Apply Now

Not SpecifiedCompensation

Senior (5 to 8 years), Expert & Leadership (9+ years)Experience Level

Full TimeJob Type

UnknownVisa

AI, TechnologyIndustries

Requirements

15+ years of software engineering experience with deep expertise in distributed systems or large-scale backend infrastructure
BS, MS, or PhD in Computer Science, Electrical/Computer Engineering, Physics, Mathematics, other Engineering or related fields (or equivalent experience)
Strong programming skills in Python, Go, or C++ with a track record of building production-grade, highly available systems
Proven experience with container orchestration technologies like Kubernetes
A deep understanding of system architecture for high-performance, low-latency API services
Experience in designing, implementing, and optimizing systems for GPU resource management
Familiarity with modern observability tools (e.g., DataDog, Prometheus, Grafana, OpenTelemetry)
Demonstrated experience with deployment strategies and CI/CD pipelines
Excellent problem-solving skills and the ability to work in a fast-paced, collaborative environment

Responsibilities

Lead the design and development of a scalable, robust, and reliable platform for serving AI models for inference as a service
Architect and implement systems for dynamic GPU resource management, autoscaling, and efficient scheduling of inference workloads
Build and maintain the core infrastructure, including load balancing and rate limiting, to ensure the stability and high availability of inference services
Define and implement APIs for model deployment, monitoring, and management for a seamless user experience
Optimize system performance and latency for various model types, from large language models (LLMs) to computer vision models, ensuring high-throughput and responsiveness
Collaborate with engineering teams to integrate deployment, monitoring, and performance telemetry into our CI/CD pipelines
Develop tools and frameworks for real-time observability, performance profiling, and debugging of inference services
Drive architectural decisions and best practices for long-term platform evolution and scalability
Contribute to NVIDIA's AI Factory initiative by building a foundational platform that supports model serving needs

Skills

Key technologies and capabilities for this role

GPUAutoscalingSchedulingLoad BalancingRate LimitingAPIsLLMComputer VisionInferenceAI Models

Questions & Answers

Common questions about this position

What is the location for this Principal Software Engineer role?

The position is based in Santa Clara, CA.

What salary or compensation is offered for this position?

This information is not specified in the job description.

What are the key skills required for this role?

Candidates need 15+ years of software engineering experience in distributed systems or large-scale backend infrastructure, strong programming skills in Python, Go, or C++, and proven experience with container orchestration technologies like Kubernetes.

What is the company culture like at NVIDIA?

NVIDIA offers a diverse, supportive environment where everyone is inspired to do their best work.

What makes a strong candidate for this Principal Software Engineer position?

A strong candidate will have 15+ years of experience in distributed systems or large-scale backend infrastructure, expertise in Python, Go, or C++, experience with Kubernetes, and a BS, MS, or PhD in Computer Science or related fields.

NVIDIA

Designs GPUs and AI computing solutions

About NVIDIA

NVIDIA designs and manufactures graphics processing units (GPUs) and system on a chip units (SoCs) for various markets, including gaming, professional visualization, data centers, and automotive. Their products include GPUs tailored for gaming and professional use, as well as platforms for artificial intelligence (AI) and high-performance computing (HPC) that cater to developers, data scientists, and IT administrators. NVIDIA generates revenue through the sale of hardware, software solutions, and cloud-based services, such as NVIDIA CloudXR and NGC, which enhance experiences in AI, machine learning, and computer vision. What sets NVIDIA apart from competitors is its strong focus on research and development, allowing it to maintain a leadership position in a competitive market. The company's goal is to drive innovation and provide advanced solutions that meet the needs of a diverse clientele, including gamers, researchers, and enterprises.

Santa Clara, CaliforniaHeadquarters

1993Year Founded

$19.5MTotal Funding

IPOCompany Stage

Automotive & Transportation, Enterprise Software, AI & Machine Learning, GamingIndustries

10,001+Employees

Benefits

Company Equity

401(k) Company Match

Risks

Increased competition from AI startups like xAI could challenge NVIDIA's market position.

Serve Robotics' expansion may divert resources from NVIDIA's core GPU and AI businesses.

Integration of VinBrain may pose challenges and distract from NVIDIA's primary operations.

Differentiation

NVIDIA leads in AI and HPC solutions with cutting-edge GPU technology.

The company excels in diverse markets, including gaming, data centers, and autonomous vehicles.

NVIDIA's cloud services, like CloudXR, offer scalable solutions for AI and machine learning.

Upsides

Acquisition of VinBrain enhances NVIDIA's AI capabilities in the healthcare sector.

Investment in Nebius Group boosts NVIDIA's AI infrastructure and cloud platform offerings.

Serve Robotics' expansion, backed by NVIDIA, highlights growth in autonomous delivery services.

Land your dream remote job 3x faster with AI

Try Jobo Free