Lead Data Engineer – Big Data

New York, New York, United States

Apply with AI Apply

$118,094 – $237,100Compensation

Expert & Leadership (9+ years)Experience Level

Full TimeJob Type

UnknownVisa

Financial Services, Data & AnalyticsIndustries

Requirements

Candidates must possess a BE, MCA, or MS degree in Computer Science or Information Technology, with over 10 years of experience as a Data Engineer in an innovative organization. Required experience includes over 5 years in implementing data lake systems using AWS/Azure cloud technologies like S3, Databricks, Hive, and large-scale data analytics tools with Scala and Python, plus 3 years of expertise in building applications using data stream processing tools for bulk and incremental data loads. Strong expertise, preferably 5 years, in the SDLC cycle including stakeholder, people, risk, strategic, and communication management is also necessary, along with hands-on experience in developing scalable data pipelines using technologies like Databricks and Scala, applying ETL and ELT concepts. Deep experience with three or more technologies from Java/J2EE, C#, AWS, Large-scale data analytics tools, Python, Scala, any RDBMS, Data stream processing tools, Informatica, Angular/ReactJS, Databricks, and Cloud-native orchestration tools is required, as is experience with CI/CD tools like Jenkins and Azure DevOps, working in a UNIX/Linux environment with shell scripting, and a strong understanding of cloud-native technologies.

Responsibilities

The Lead Data Engineer will design and build ingestion pipelines using ELT and schema on read in Databricks Delta Lake, and develop transformations at scale using ELT frameworks. They will provide technical expertise in the design and implementation of Ratings Data Ingestion pipelines using modern AWS cloud technologies such as S3, Hive, Databricks, Scala, Python, and large-scale data analytics tools. Responsibilities include building and maintaining a data environment for speed, accuracy, consistency, and uptime, collaborating with other data teams and Data Science teams on ingestion pipeline development, and ensuring data governance principles, data quality checks, and data lineage are implemented. The role also involves staying current with emerging Big Data and cloud technologies, evaluating new technologies, and ensuring compliance through the adoption of enterprise standards and promotion of best practices aligned with organizational standards.

Skills

Data Engineering

Big Data

ETL

ELT

Databricks

Delta Lake

AWS

Hive

Scala

Python

Data Pipelines

Data Governance

Data Quality

Data Lineage

Cloud Technologies

Schema on Read

S&P Global

Provides financial information and analytics services

About S&P Global

S&P Global provides financial information and analytics to a wide range of clients, including investors, corporations, and governments. The company offers services such as credit ratings, market intelligence, and indices, which help clients understand and navigate the global financial market. S&P Global's products work by utilizing advanced data analytics and research to deliver insights that assist clients in making informed decisions and managing risks. Unlike many competitors, S&P Global has a diverse range of divisions, including S&P Global Ratings and S&P Dow Jones Indices, which allows it to cater to various financial needs. The company's goal is to support clients in driving growth while also committing to corporate responsibility and positive societal impact.

New York City, New YorkHeadquarters

1917Year Founded

IPOCompany Stage

Data & Analytics, Financial ServicesIndustries

10,001+Employees

Benefits

Health Insurance

Unlimited Paid Time Off

Professional Development Budget

401(k) Company Match

Family Planning Benefits

Employee Discounts

Risks

Integration challenges with new acquisitions like ProntoNLP may cause operational issues.

Increased competition from AI-driven platforms like Brooklyn Investment Group.

Dependence on volatile credit ratings market could impact revenue stability.

Differentiation

S&P Global integrates advanced AI tools for superior financial analytics capabilities.

The company offers comprehensive ESG solutions, meeting growing sustainability demands.

S&P Global's diverse divisions provide a wide range of financial services globally.

Upsides

Acquisition of ProntoNLP boosts data analytics and sentiment scoring capabilities.

Rising demand for ESG data enhances S&P Global's market position.

Expansion into India strengthens S&P Global's research and insights offerings.

Land your dream remote job 3x faster with AI

Try Jobo Free