Data Pipeline Engineer

India

Apply with AI Apply

Not SpecifiedCompensation

Mid-level (3 to 4 years), Senior (5 to 8 years)Experience Level

Full TimeJob Type

UnknownVisa

Market Research, Analytics, Financial Services, BiotechnologyIndustries

Requirements

Candidates should have 2-4 years of data engineering experience, with a strong understanding of Spark and SQL. Prior data pipeline experience is required, and familiarity with Databricks, Spark, and AWS is preferred.

Responsibilities

The Data Pipeline Engineer will design, build, and maintain reliable data pipelines, detect and resolve performance bottlenecks using Spark UI, logs, and metrics, and contribute ideas to improve data models and ETL workflows. They will collaborate with analysts and engineers to convert business requirements into scalable datasets, participate in code reviews, author documentation, and share learnings through demos and write-ups.

Skills

Data Pipelines

ETL

Data Warehousing

SQL

Python

Cloud Platforms (AWS, GCP, Azure)

Big Data Technologies (Spark, Hadoop)

Data Modeling

Data Quality

Performance Optimization

YipitData

Data analysis and research services provider

About YipitData

YipitData provides data analysis and research services by converting large amounts of raw data from various sources into clear and actionable insights. These insights help clients, primarily investors and corporate entities, understand market trends and company performance. YipitData operates on a subscription model, offering clients regular datasets and reports that include updates on various sectors such as autos, marketplaces, and logistics. Each subscription provides tailored information with three layers of coverage to meet specific client needs. The company differentiates itself by offering deep dives into topics, continuous tracking of metrics, and data integration best practices, ensuring clients have access to relevant and detailed information. The goal of YipitData is to empower clients to make informed decisions based on comprehensive data analysis.

New York City, New YorkHeadquarters

2013Year Founded

$561.3MTotal Funding

SERIES_ECompany Stage

Data & Analytics, Enterprise Software, Financial ServicesIndustries

501-1,000Employees

Benefits

401(k) Company Match

Flexible Work Hours

Unlimited Paid Time Off

Parental Leave

Wellness Program

Professional Development Budget

Risks

Increased competition from AI-driven analytics platforms like ChatGPT.

Shift towards e-commerce may reduce demand for traditional market research services.

Inflation volatility could impact accuracy and demand for predictive analytics products.

Differentiation

YipitData specializes in alternative data, offering unique insights beyond traditional data sources.

The company provides custom reports tailored to specific business needs and sectors.

YipitData's subscription model ensures clients receive regular, detailed datasets and reports.

Upsides

Expansion into Europe and China opens new markets for localized data insights.

Carlyle Group's investment provides capital for technology and market expansion.

Focus on tracking inflation through alternative data offers a unique client value proposition.

Land your dream remote job 3x faster with AI

Try Jobo Free