Data Engineer, DaaS

Position Overview

• Location Type: Remote
• Employment Type: Full-time
• Salary: Not specified

PayPay is a FinTech company with over 65M users as of August 2024. Our team is diverse, with members from over 50 countries. Our vision is unlimited – we dare to believe we don't need a clear vision to create a future beyond our imagination. We are looking for people who can embrace challenges, refresh our product at breakneck speed, and promote PayPay with professionalism and passion.

Job Description

PayPay’s growth is driving a rapid expansion of PayPay product teams, and the need for a robust Data Engineering Platform is more critical than ever. The DaaS team’s responsibility is to design, implement, and operate this platform using cutting-edge technologies such as Spark, Databricks, Delta Lake, PySpark, Scala, and AWS suite of tools. We are looking for talented Data Engineers to join our team and help us scale our platform across the organization.

Main Responsibilities

• Design, develop, and maintain scalable data ingestion pipelines using Databricks, Airflow, Lambda, and Terraform.
• Optimize and manage large-scale data pipelines to ensure high performance, reliability, and efficiency.
• Implement data processing workflows using Spark, Delta Lake, PySpark, and Scala.
• Maintain and enhance Data Lakehouse with Unity Catalog for effective data management and discovery.
• Build automation, framework, and improved tooling to support end-to-end data engineering workflows.
• Collaborate with cross-functional teams to ensure seamless data flow and integration across the organization.
• Implement best practices for observability, data governance, security, and compliance.

Qualifications

• Experience: 5+ years experience as a Data Engineer or in a similar role.
• Core Technologies: Hands-on experience with Databricks, Delta Lake, Spark, and Scala.
• Data Architecture: Experience designing, building, and operating a Data Lake or Data Warehouse.
• Orchestration: Knowledge of Data Orchestration tools such as Airflow, Dagster, or Prefect.
• CDC: Familiarity with change data capture tools like Canal, Debezium, and Maxwell.
• Programming: Proficiency in at least one primary language (e.g., Scala, Python) and SQL (any variant).
• Data Cataloging: Experience with data cataloging and metadata management using AWS Glue Data Catalog, Lakeformation, or Unity Catalog.
• IaC: Proficiency in Terraform for infrastructure as code (IaC).
• Skills: Strong problem-solving and debugging skills to troubleshoot complex data issues.
• Collaboration: Excellent communication and collaboration skills.

Application Instructions

• Not specified.