Not SpecifiedCompensation
Mid-level (3 to 4 years)Experience Level
Full TimeJob Type
UnknownVisa
Non-profit, Biotechnology, Information ServicesIndustries

Requirements

Candidates should have 3+ years of data engineering experience with exposure to on-premise systems like Spark, Hadoop, and HDFS. A strong understanding of engineering best practices, maintainable code, and troubleshooting systems for performance and scaling is required. Desirable qualifications include experience with architectural/system design, technical ownership, data governance, data lineage, and data quality initiatives. Proficiency in Python or Java/Scala, knowledge of SQL and various database/query dialects, and working knowledge of CI/CD processes and software containerization are essential. Familiarity with stream processing frameworks, Kubernetes, Flink, Iceberg, Druid, Presto, Cassandra, and AI development tooling are considered bonus skills. Good communication and collaboration skills are also necessary.

Responsibilities

The Data Engineer will design and build scalable, robust data pipelines using tools such as Airflow, Spark, and Kafka. Responsibilities include implementing monitoring and alerting systems for data quality, assisting in the design and implementation of solutions for data governance and lineage, and collaborating with peers to improve and evolve the shared data platform. The role also involves enhancing operational excellence by identifying and implementing improvements in system reliability, maintainability, and performance.

Skills

Data Engineering
Spark
Hadoop
HDFS
Airflow
Kafka
Hive
Python
Java
Scala
SQL
MariaDB
HiveQL
CassandraQL
Spark SQL
Presto
CI/CD
Software Containerization
Kubernetes
Flink
Iceberg
Druid
Cassandra
Data Governance
Data Lineage
Data Quality

Wikimedia Foundation

Operates Wikipedia and free knowledge projects

About Wikimedia Foundation

The Wikimedia Foundation operates Wikipedia and other free knowledge projects, aiming to create a world where everyone can freely access and share knowledge. It provides a platform for users to read, contribute, and share content, while also supporting the volunteer communities that help maintain these projects. The foundation is funded by donations from individuals and institutions, emphasizing its nonprofit status. Unlike many other organizations, it focuses on making knowledge accessible to all without charge, advocating for policies that support free knowledge initiatives. Its goal is to empower individuals to contribute to and benefit from a collective pool of knowledge.

San Francisco, CaliforniaHeadquarters
2003Year Founded
$145.9MTotal Funding
GRANTCompany Stage
Social Impact, EducationIndustries
501-1,000Employees

Benefits

Remote Work Options

Risks

Reliance on Nvidia's AI tech may affect Wikimedia's data processing autonomy.
DSA audit could reveal vulnerabilities requiring significant resources to address.
Decentralized platforms like Mastodon may divert users from Wikipedia.

Differentiation

Wikimedia Foundation operates the world's largest free online encyclopedia, Wikipedia.
It supports a diverse range of projects like Wiktionary and Wikisource.
The Foundation is a non-profit, relying on global donations for funding.

Upsides

Nvidia's NeMo Retriever tech reduced Wikipedia processing time from 30 days to 3 days.
Holistic AI's audit under the DSA enhances Wikimedia's platform safety and accountability.
Collaboration with Open Foundation West Africa combats misinformation during Ghana's elections.

Land your dream remote job 3x faster with AI