Data Engineer (Web Scraping)

India

Apply with AI Apply

Not SpecifiedCompensation

Mid-level (3 to 4 years), Senior (5 to 8 years)Experience Level

Full TimeJob Type

UnknownVisa

Market Research, Analytics, Financial Services, BiotechnologyIndustries

Requirements

Candidates should have experience in designing, building, and maintaining web scrapers. Familiarity with advanced scraping techniques such as fingerprinting methods (cookies, headers, user-agent rotation, proxies) is required. Experience handling dynamic content, complex DOM structures, and managing session/cookie lifecycles is also necessary. The role requires the ability to develop monitoring solutions and alerting frameworks. Staying up-to-date with industry trends and evolving bot-detection tactics is important.

Responsibilities

The Web Scraping Engineer will be responsible for refactoring and maintaining existing web scrapers to improve reliability and efficiency, implementing best coding practices, and utilizing advanced techniques to avoid detection. This role involves collaborating with cross-functional teams to gather requirements and ensure data quality, as well as providing support and documentation to internal stakeholders. Additionally, the engineer will monitor and troubleshoot scraper performance, diagnose bottlenecks, and drive continuous improvement by proposing new tooling and methodologies.

Skills

web scraping

data analysis

Python

SQL

data engineering

market research

alternative data

YipitData

Data analysis and research services provider

About YipitData

YipitData provides data analysis and research services by converting large amounts of raw data from various sources into clear and actionable insights. These insights help clients, primarily investors and corporate entities, understand market trends and company performance. YipitData operates on a subscription model, offering clients regular datasets and reports that include updates on various sectors such as autos, marketplaces, and logistics. Each subscription provides tailored information with three layers of coverage to meet specific client needs. The company differentiates itself by offering deep dives into topics, continuous tracking of metrics, and data integration best practices, ensuring clients have access to relevant and detailed information. The goal of YipitData is to empower clients to make informed decisions based on comprehensive data analysis.

New York City, New YorkHeadquarters

2013Year Founded

$561.3MTotal Funding

SERIES_ECompany Stage

Data & Analytics, Enterprise Software, Financial ServicesIndustries

501-1,000Employees

Benefits

401(k) Company Match

Flexible Work Hours

Unlimited Paid Time Off

Parental Leave

Wellness Program

Professional Development Budget

Risks

Increased competition from AI-driven analytics platforms like ChatGPT.

Shift towards e-commerce may reduce demand for traditional market research services.

Inflation volatility could impact accuracy and demand for predictive analytics products.

Differentiation

YipitData specializes in alternative data, offering unique insights beyond traditional data sources.

The company provides custom reports tailored to specific business needs and sectors.

YipitData's subscription model ensures clients receive regular, detailed datasets and reports.

Upsides

Expansion into Europe and China opens new markets for localized data insights.

Carlyle Group's investment provides capital for technology and market expansion.

Focus on tracking inflation through alternative data offers a unique client value proposition.

Land your dream remote job 3x faster with AI

Try Jobo Free