Requirements

Candidates must be proficient in Python and backend development, with experience in large codebases. They should have experience building and operating large-scale stream and batch processing pipelines using technologies such as Kafka, Spark, Flink, and Presto/Trino. Hands-on experience with Kubernetes and Terraform, as well as deploying and troubleshooting production systems, is required. Experience working on access control, provenance, auditing, and large-scale data movement is essential. A passion for building systems that provide key insights, especially in ML training workflows, is necessary, along with the ability to thrive in a fast-moving environment while making impactful trade-offs. Understanding data transformations in ML training and inference workflows is a plus.