Data Engineer

Employment Type: Full-time Location Type: On-site Salary: Not specified

Position Overview

Responsible for assembling large, complex sets of data that meet non-functional and functional business requirements. This role involves identifying, designing, and implementing internal process improvements, including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes. The Data Engineer will build the required infrastructure for optimal extraction, transformation, and loading of data from various data sources using Azure, Databricks, and SQL technologies.

Responsibilities

• Transform conceptual algorithms from R&D into efficient, production-ready code.
• Possess a strong mathematical background to document and maintain code.
• Integrate finished models into larger data processes using UNIX scripting languages such as ksh, Python, Spark, Scala, etc.
• Produce and maintain documentation for released data sets, new programs, shared utilities, or static data, adhering to department standards.
• Ensure quality deliverables to clients by following existing quality processes, manually calculating comparison data, developing statistical pass/fail testing, and visually inspecting data for reasonableness, aiming for on-time delivery with zero defects.
• Identify, design, and implement internal process improvements, including infrastructure re-design for scalability, data delivery optimization, and automation of manual processes.
• Build and maintain infrastructure for optimal extraction, transformation, and loading (ETL) of data from various sources using Azure, Databricks, and SQL technologies.

Qualifications

Education/Training

• B.E./B.Tech. with a major in Computer Science, BIS, CIS, Electrical Engineering, Operations Research, or another technical field.
• Coursework or experience in Numerical Analysis, Mathematics, or Statistics is a plus.

Hard Skills

• Proven experience working as a data engineer.
• Highly proficient in using the Spark framework (Python and/or Scala).
• Extensive knowledge of Data Warehousing concepts, strategies, and methodologies.
• Programming experience in Python, SQL, Scala.
• Direct experience building data pipelines using Apache Spark (preferably in Databricks) and Airflow.
• Hands-on experience designing and delivering solutions using Azure, including Azure Storage, Azure SQL Data Warehouse, and Azure Data Lake.
• Experience with big data technologies (Hadoop).
• Ability to write highly efficient code in terms of performance and memory utilization.
• Basic knowledge of SQL; capable of handling common functions.

Experience

• Minimum 3-6 years of experience as a Data Engineer.
• Experience modeling or manipulating large amounts of data is a plus.
• Experience with Demographic or Retail business is a plus.

Soft Skills

• Must be team-oriented with strong collaboration, prioritization, and adaptability skills.

Certifications (Plus)

• Databricks & Azure Big Data Architecture Certification.

Company Information

About NIQ

NIQ is the world’s leading consumer intelligence company, delivering the most complete understanding of consumer buying behavior and revealing new pathways to growth. In 2023, NIQ combined with GfK, bringing together the two industry leaders with unparalleled global reach. With a holistic retail read and the most comprehensive consumer insights—delivered with advanced analytics through state-of-the-art platforms—NIQ delivers the Full View™. NIQ is an Advent International portfolio company with operations in 100+ markets, covering more than 90% of the world’s population.

For more information, visit NIQ.com.

Our Commitment to Diversity, Equity, and Inclusion

NIQ is committed to reflecting the diversity of the clients, communities, and markets we measure within our own workforce. We exist to count everyone and are on a mission to systematically embed inclusion and diversity into all aspects of our workforce, measurement, and products.

Our Benefits

• Flexible working environment
• Volunteer time off
• LinkedIn Learning
• Employee-Assistance-Program (EAP)

Follow Us

• LinkedIn
• Instagram
• Twitter
• Facebook