Staff Software Engineer, AI Gateway

United States

Apply with AI Apply

$200,400 – $275,600Compensation

Expert & Leadership (9+ years)Experience Level

Full TimeJob Type

UnknownVisa

Software Development, Artificial Intelligence, Cloud ComputingIndustries

Requirements

Candidates should possess 8+ years of backend engineering experience with production-grade systems, demonstrating deep knowledge of distributed and highly scalable systems, cloud-native infrastructure, and API design. Experience building secure, high-throughput services such as gateways, proxies, load balancers, or policy engines is essential. Fluency in Go and/or Rust is required, with both being preferred. Familiarity with AI/ML platforms or model serving infrastructure, along with a strong product mindset and an ownership mentality with a bias for shipping, learning, and iterating, are also key qualifications. Bonus qualifications include prior experience with LLM APIs, RAG architectures, vector databases, agent frameworks, policy engines, observability frameworks, API gateways, OAuth2.1, secret management, cloud IAM systems, Kubernetes, Docker, and microservices architecture.

Responsibilities

The Staff Software Engineer will design and implement core systems for the AI Gateway, including the model router, MCP gateway, and control plane. They will build infrastructure supporting dynamic model selection, auto-failover, cost-based routing, and policy enforcement. Responsibilities include owning critical capabilities like secure credential storage, session summarization, caching, and rate limiting. The engineer will develop APIs for developers using OpenAI-compatible interfaces and the Model Context Protocol, and build underlying infrastructure for evaluation, telemetry, replay, and backtesting of agents and LLM workflows. Additionally, they will lead architectural decisions, mentor engineers, collaborate with product and design on the control plane UI, and contribute to roadmap planning, technical strategy, and cross-functional alignment. Key problems to solve involve building a unified abstraction layer across model and tool providers, implementing secure and scalable identity and credential vaulting, creating infrastructure for real-time and historical analytics of AI agent behavior, and ensuring end-to-end policy enforcement and logging.

Skills

AI Gateway

Model Orchestration

Tool Management

Observability

Governance

OpenAI-compatible interface

MCP server

Auto-RAG

Tool Injection

Session Summarization

Cost Optimization

API Development

Credential Storage

Rate Limiting

Evaluation

Telemetry

Backtesting

LLM Workflows

System Design

Infrastructure Development

Cloud Computing

Distributed Systems

Docker

Containerization platform for application development

About Docker

Docker offers a containerization platform that enables developers to build, share, and run applications in isolated environments called containers. These containers package applications with their dependencies, ensuring consistent performance across different stages of development and deployment. Docker stands out from competitors by simplifying the development process and enhancing collaboration, with a freemium business model that includes various subscription tiers and services like Docker Hub and Docker Desktop. The company's goal is to streamline software development and deployment, making it easier for developers to manage applications efficiently.

Palo Alto, CaliforniaHeadquarters

2013Year Founded

$484.3MTotal Funding

SERIES_CCompany Stage

Consumer Software, Enterprise SoftwareIndustries

501-1,000Employees

Benefits

Flexible Work Hours

Home Office Stipend

Parental Leave

Phone/Internet Stipend

Unlimited Paid Time Off

Professional Development Budget

Company Equity

Health Insurance

Risks

Competition from Kubernetes could impact Docker's market share.

Alternative container runtimes like Podman pose a threat to Docker's dominance.

Security vulnerabilities in Docker containers could undermine platform trust.

Differentiation

Docker offers a comprehensive platform for building, sharing, and running applications.

Docker's containerization ensures consistency across development and deployment stages.

Docker's freemium model provides flexibility with various subscription tiers.

Upsides

Increased adoption of Kubernetes drives demand for Docker's containerization solutions.

Docker's acquisitions enhance its performance, flexibility, and security features.

Growing hybrid and multi-cloud environments position Docker as a key player.

Land your dream remote job 3x faster with AI

Try Jobo Free