Senior Software Engineer, Data Infrastructure
RoSalary not specified
- Full Time
- Senior (5 to 8 years)
$295,000 – $440,000Compensation
Mid-level (3 to 4 years), Senior (5 to 8 years)Experience Level
Full TimeJob Type
UnknownVisa
Data & Analytics, Enterprise Software, AI & Machine LearningIndustries
Requirements
Candidates must be proficient in Python and backend development, with experience in large codebases. They should have experience building and operating large-scale stream and batch processing pipelines using technologies such as Kafka, Spark, Flink, and Presto/Trino. Hands-on experience with Kubernetes and Terraform, as well as deploying and troubleshooting production systems, is required. Experience working on access control, provenance, auditing, and large-scale data movement is essential. A passion for building systems that provide key insights, especially in ML training workflows, is necessary, along with the ability to thrive in a fast-moving environment while making impactful trade-offs. Understanding data transformations in ML training and inference workflows is a plus.
Responsibilities
The Software Engineer will build and maintain large-scale stream and batch processing pipelines. They will develop a general-purpose data processing platform for handling massive datasets and scale applications for ML research. Ensuring the security, integrity, and compliance of data according to industry standards is critical. The engineer will ensure the analytics and data platforms can scale reliably and accelerate company productivity by empowering engineers and researchers with excellent data tooling. They will collaborate with product engineers and other teams to build technical foundations and participate in an on-call rotation to respond to critical incidents.
Skills
Kafka
Kubernetes
Presto
Trino
Flink
Distributed Systems
Data Pipelines
Stream Processing
Batch Processing
SQL
OpenAI
Develops safe and beneficial AI technologies
About OpenAI
OpenAI develops and deploys artificial intelligence technologies aimed at benefiting humanity. The company creates advanced AI models capable of performing various tasks, such as automating processes and enhancing creativity. OpenAI's products, like Sora, allow users to generate videos from text descriptions, showcasing the versatility of its AI applications. Unlike many competitors, OpenAI operates under a capped profit model, which limits the profits it can make and ensures that excess earnings are redistributed to maximize the social benefits of AI. This commitment to safety and ethical considerations is central to its mission of ensuring that artificial general intelligence (AGI) serves all of humanity.
San Francisco, CaliforniaHeadquarters
2015Year Founded
$18,433.2MTotal Funding
LATE_VCCompany Stage
AI & Machine LearningIndustries
1,001-5,000Employees