Sr. Data Engineer

United States

Apply with AI Apply

Not SpecifiedCompensation

Senior (5 to 8 years)Experience Level

Full TimeJob Type

UnknownVisa

AutomotiveIndustries

Requirements

Candidates must have 5+ years of experience with object-oriented/object function scripting languages like Python, Java, C++, or Scala. A Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field, or equivalent experience, is required. Strong, hands-on experience with Databricks data engineering and management is essential. Experience setting up CI/CD environments in Azure and Databricks using tools like Azure DevOps, ARM templates, GitHub Actions, DBX Asset Bundles, and Terraform is necessary. A strong understanding of Gen AI tools and their integration with daily tasks in Databricks, such as Microsoft Copilot and Databricks Genie, is required. Experience with big data frameworks like Apache Hadoop, Apache Spark, or Apache Kafka, and expertise in designing streaming data pipelines using FiveTran, Azure Event Hubs, Auto Loader, and Delta Lake in Azure Databricks are crucial. Hands-on experience with data serialization formats such as JSON, Parquet, YML, and XML is also required. At least 3 years of hands-on experience with Big Data Tools like Hadoop, Spark, and Kafka is necessary. A demonstrable understanding of networking/distributed computing concepts is required.

Responsibilities

The Data Engineer will build industrialized data assets and optimize data pipelines to support Business Intelligence and Advanced Analytic objectives. They will work closely with Data Scientists, BI Developers, System Architects, and Data Architects to deliver value. Responsibilities include communicating and maintaining Master Data, Metadata, Data Management Repositories, Logical Data Models, and Data Standards. They will create and maintain optimal data pipeline architecture, assemble large, complex data sets, and identify and implement internal process improvements such as automation and infrastructure optimization. The role involves building industrialized analytic datasets and delivery mechanisms to provide actionable insights and working with business partners on data-related technical issues to develop requirements. They will also design data solutions that are highly automated and provide consistent, accurate outcomes, and orchestrate automated workflows or batch jobs on the Azure Databricks platform. Understanding of data governance principles, data privacy regulations, and implementing security measures for data protection is expected, as is the ability to work effectively in cross-functional teams.

Skills

Python

Java

C++

Scala

Data Engineering

Databricks

CI/CD

Azure

Data Pipelines

Master Data Management

Metadata Management

Data Modeling

Data Standards

Business Intelligence

Advanced Analytics

General Motors

Designs, manufactures, and sells vehicles

About General Motors

General Motors designs, manufactures, and sells vehicles and vehicle parts, catering to individual consumers, businesses, and government entities. The company operates in both traditional internal combustion engine vehicles and the growing electric vehicle (EV) market, generating revenue through vehicle sales and financing services. GM stands out from competitors with its commitment to community service, sustainability, and diversity, as evidenced by a majority female Board of Directors. The company's goal is to balance traditional automotive manufacturing with technological advancements in electric and autonomous vehicles.

Detroit, MichiganHeadquarters

1908Year Founded

$486.7MTotal Funding

IPOCompany Stage

Automotive & Transportation, Financial ServicesIndustries

10,001+Employees