AI/HPC Network Development Engineer - Networking at xAI

Palo Alto, California, United States

Apply Now

Not SpecifiedCompensation

Senior (5 to 8 years), Expert & Leadership (9+ years)Experience Level

Full TimeJob Type

UnknownVisa

AI, HPC, TechnologyIndustries

Requirements

A minimum of 10 years designing and operating large scale networks with 5 years in the ethernet AI/HPC space
Deep understanding of congestion control on ethernet with Infiniband an added bonus
Deep understanding of AI training and inference workloads and how they operate on the network, including ability to use and debug NCCL and potentially commit to the library
Expertise in creating a portfolio of metrics for performance and operations to optimize the fleet for training and inference traffic
Experience with Python to automate away repetitive tasks and facilitate daily job working with and analyzing large sets of data
Deep experience in RoCEv2
Strong communication skills to concisely and accurately share knowledge with teammates
Willingness to travel significantly to Memphis for data center buildouts and to Palo Alto for team collaboration
Ability to participate in a team on-call rotation and contribute to scaling and maintenance efforts

Responsibilities

Develop network at hyper scale while optimizing performance and availability with deep experience in RoCEv2
Analyze and understand current network performance and availability, then optimize it for training models and customer inference queries
Spend time deep inside NCCL, building metric dashboards, and tweaking configurations to maximize performance
Help design the next iteration of backend and front-end networks to enable seamless build-out of new GPU infrastructure with minimal engineering assistance
Participate in significant travel to Memphis for building capacity
Participate in team on-call rotation and assist with scaling and maintenance efforts
Contribute to deployment and operations frameworks to eliminate repetitive tasks

Skills

Key technologies and capabilities for this role

RoCEv2NCCLEthernetGPU ClustersHPC NetworkingPerformance OptimizationMetric DashboardsNetwork ConfigurationScaling InfrastructureOn-call Rotation

Questions & Answers

Common questions about this position

What are the location options and travel requirements for this role?

There are two openings: one based in Palo Alto, California and the other in Dublin, Ireland, with significant travel expected to Memphis, Tennessee for data center buildouts and to the head office in Palo Alto for team collaboration.

What are the required qualifications and experience for this position?

Candidates need a minimum of 10 years designing and operating large scale networks with 5 years in ethernet AI/HPC space, deep understanding of congestion control on ethernet (Infiniband a bonus), AI training/inference workloads with NCCL debugging, expertise in performance metrics, and Python experience for automation.

What is the company culture like at xAI?

xAI has a small, highly motivated team focused on engineering excellence, operating with a flat organizational structure where employees are hands-on, show initiative, and contribute directly to the mission, with emphasis on work ethic, prioritization, and strong communication skills.

Is this a remote position, or are there office requirements?

The roles are based in Palo Alto, California or Dublin, Ireland, with significant travel to Memphis and Palo Alto, but no mention of full remote work.

What makes a strong candidate for this AI/HPC Network Development Engineer role?

A strong candidate has 10+ years in large-scale networks including 5+ in ethernet AI/HPC, deep expertise in RoCEv2 congestion control, NCCL debugging for AI workloads, metrics for optimization, and Python automation skills, plus the ability to thrive in a hands-on environment with travel and on-call duties.

xAI

AI tools for research and information retrieval

About xAI

x.ai develops AI tools aimed at enhancing research and information retrieval. Their main product, Grok, is designed to answer a variety of questions, including unconventional ones that other AI systems might not handle. Grok provides real-time knowledge, making it a useful resource for researchers, academics, and professionals who need quick access to relevant information. Unlike competitors, Grok stands out for its ability to suggest questions and provide nuanced answers, catering to a diverse range of inquiries. The goal of x.ai is to empower users by streamlining their research processes and fostering innovation through reliable information access.

Burlingame, CaliforniaHeadquarters

2023Year Founded

$11,803.1MTotal Funding

SERIES_CCompany Stage

Data & Analytics, AI & Machine LearningIndustries

1,001-5,000Employees

Benefits

Health Insurance

Remote Work Options

Risks

Increased competition from Anthropic could challenge xAI's market position.

Legal battles involving Elon Musk may divert resources from xAI's operations.

Reliance on Nvidia GPUs poses risks if supply chain issues arise.

Differentiation

Grok answers unconventional questions, unlike many AI systems.

xAI's Grok provides real-time knowledge, enhancing research efficiency.

Grok's ability to generate striking images sets it apart in visual data processing.

Upsides

xAI secured $6 billion funding, boosting AI infrastructure and R&D.

Grok's iOS app launch expands user accessibility and engagement.

AI-driven research tools are increasingly integrated with cloud platforms, aiding xAI's growth.

Land your dream remote job 3x faster with AI

Try Jobo Free