Responsibilities

The Data Engineer will be responsible for designing, developing, and maintaining scalable data ingestion pipelines using Databricks, Airflow, Lambda, and Terraform, optimizing and managing large-scale data pipelines for high performance and reliability, implementing data processing workflows using Spark, Delta Lake, PySpark, and Scala, maintaining and enhancing a Data Lakehouse with Unity Catalog for effective data management and discovery, building automation, framework, and improved tooling to support end-to-end data engineering workflows, collaborating with cross-functional teams to ensure seamless data flow and integration, and implementing best practices for observability, data governance, security, and compliance.