Staff Big Data Engineer
H1Full Time
Expert & Leadership (9+ years)
Candidates should have at least 3 years of data engineering experience, including exposure to on-premise systems like Spark, Hadoop, and HDFS. A strong understanding of engineering best practices, proficiency in Python or Java/Scala, knowledge of SQL and various database/query dialects, and working knowledge of CI/CD processes and software containerization are essential. Desirable qualifications include experience with architectural/system design, technical ownership, data governance, data lineage, and data quality initiatives. Familiarity with stream processing frameworks, Kubernetes, Flink, Iceberg, Druid, Presto, Cassandra, and AI development tooling is also beneficial. Good communication and collaboration skills are required to interact effectively within and across teams and articulate ideas to both technical and non-technical stakeholders.
The Data Engineer will design and build scalable, robust data pipelines using tools such as Airflow, Spark, and Kafka. They will implement monitoring and alerting systems for data quality, and support data governance and lineage initiatives by designing and implementing solutions to track and manage data across pipelines. The role involves collaborating with peers to improve and evolve the shared data platform for various use cases and enhancing operational excellence by identifying and implementing improvements in system reliability, maintainability, and performance.
Operates Wikipedia and free knowledge projects
The Wikimedia Foundation operates Wikipedia and other free knowledge projects, aiming to create a world where everyone can freely access and share knowledge. It provides a platform for users to read, contribute, and share content, while also supporting the volunteer communities that help maintain these projects. The foundation is funded by donations from individuals and institutions, emphasizing its nonprofit status. Unlike many other organizations, it focuses on making knowledge accessible to all without charge, advocating for policies that support free knowledge initiatives. Its goal is to empower individuals to contribute to and benefit from a collective pool of knowledge.