Data Engineer

Remote

Apply with AI Apply

Not SpecifiedCompensation

Mid-level (3 to 4 years)Experience Level

Full TimeJob Type

UnknownVisa

Non-profit, Biotechnology, Information ServicesIndustries

Requirements

Candidates should have 3+ years of data engineering experience, including exposure to on-premise systems like Spark, Hadoop, and HDFS. A strong understanding of engineering best practices, maintainable code, and troubleshooting systems for performance and scaling is required. Experience with data pipeline tools such as Airflow, Kafka, Spark, and Hive, proficiency in Python or Java/Scala, knowledge of SQL and various database dialects, and working knowledge of CI/CD processes and software containerization are essential. Familiarity with stream processing frameworks, Kubernetes, Flink, Iceberg, Druid, Presto, Cassandra, and AI development tooling are considered bonus skills. Good communication and collaboration skills, along with the ability to produce clear technical designs and articulate ideas, are also necessary.

Responsibilities

The Data Engineer will design and build scalable, robust data pipelines using tools like Airflow, Spark, and Kafka. They will implement monitoring and alerting systems for data quality, and support data governance and lineage initiatives by assisting in the design and implementation of solutions to track and manage data. Responsibilities also include collaborating with peers to improve the shared data platform, enabling use cases like product analytics, bot detection, and image classification, and enhancing operational excellence by identifying and implementing improvements in system reliability, maintainability, and performance.

Skills

Spark

Hadoop

HDFS

Airflow

Kafka

Hive

Python

Java

Scala

SQL

MariaDB

HiveQL

CassandraQL

Spark SQL

Presto

CI/CD

Kubernetes

Flink

Iceberg

Druid

Data Governance

Data Lineage

Data Quality

Wikimedia Foundation

Operates Wikipedia and free knowledge projects

About Wikimedia Foundation

The Wikimedia Foundation operates Wikipedia and other free knowledge projects, aiming to create a world where everyone can freely access and share knowledge. It provides a platform for users to read, contribute, and share content, while also supporting the volunteer communities that help maintain these projects. The foundation is funded by donations from individuals and institutions, emphasizing its nonprofit status. Unlike many other organizations, it focuses on making knowledge accessible to all without charge, advocating for policies that support free knowledge initiatives. Its goal is to empower individuals to contribute to and benefit from a collective pool of knowledge.

San Francisco, CaliforniaHeadquarters

2003Year Founded

$145.9MTotal Funding

GRANTCompany Stage

Social Impact, EducationIndustries

501-1,000Employees