Lead Data Engineer – Python & GCP at Egen

Hyderabad, Telangana, India

Apply Now

Not SpecifiedCompensation

Senior (5 to 8 years), Expert & Leadership (9+ years)Experience Level

Full TimeJob Type

UnknownVisa

TechnologyIndustries

Requirements

10 to 12 years of hands-on experience in Python for backend or data engineering projects
Strong understanding and working experience with GCP cloud services (especially Dataflow, BigQuery, Cloud Functions, Cloud Composer, etc.)
Solid understanding of data pipeline architecture, data integration, and transformation techniques
Experience in version control systems like GitHub and knowledge of CI/CD practices
Experience in Apache Spark, Kafka, Redis, FastAPI, Airflow, GCP Composer DAGs
Strong experience in SQL with at least one enterprise database (SQL Server, Oracle, PostgreSQL, etc.)
Experience in data migrations from on-premise data sources to Cloud platforms
Bachelor's degree in Computer Science, a related field, or equivalent experience
Excellent problem-solving and analytical skills
Strong communication skills and ability to collaborate in a team environment

Responsibilities

Design, develop, test, and maintain scalable ETL data pipelines using Python
Architect enterprise solutions with technologies like Kafka, multi-cloud services, auto-scaling using GKE, Load balancers, APIGEE proxy API management, DBT, LLMs, redaction of sensitive information, DLP
Work extensively on GCP services: Dataflow, Cloud Functions, BigQuery, Cloud Composer, GCS, IAM, Cloud Run
Perform data ingestion from various sources and apply transformation and cleansing logic
Implement and enforce data quality checks, validation rules, and monitoring
Collaborate with data scientists, analysts, and other engineering teams
Manage version control using GitHub and participate in CI/CD pipeline deployments
Write complex SQL queries for data extraction and validation
Document pipeline designs, data flow diagrams, and operational support procedures
Gather requirements, conduct discovery phase, design and architect solutions, provide post-delivery support

Skills

Key technologies and capabilities for this role

PythonGCPDataflowBigQueryCloud ComposerAirflowGoogle Cloud StorageCloud FunctionsCloud RunGKEKafkaApache SparkFastAPIDBTMongoDBRedisBigtableAPIGEE

Questions & Answers

Common questions about this position

What experience level is required for this Lead Data Engineer role?

The position requires 10 to 12 years of relevant IT experience, with hands-on experience in Python for backend or data engineering projects.

What are the key technical skills needed for this position?

Key skills include strong Python experience, GCP services like Dataflow, BigQuery, Cloud Functions, and Cloud Composer, Apache Spark, Kafka, data pipeline architecture, GitHub, and CI/CD practices.

Is this a remote position or does it require office work?

This information is not specified in the job description.

What is the salary or compensation for this role?

This information is not specified in the job description.

What makes a strong candidate for this Lead Data Engineer position?

A strong candidate has 10-12 years of Python data engineering experience, deep GCP expertise including Dataflow and BigQuery, skills in Spark, Kafka, and Airflow, plus experience architecting scalable ETL/ELT pipelines and collaborating with teams.

Egen

Provides API strategies and cloud solutions

About Egen

Egen Solutions specializes in API strategies and cloud solutions to help businesses modernize and scale. They offer two main paths for application development: Rebuild, which improves existing functionalities, and Replace, which creates new applications in the cloud. Egen stands out by providing tailored digital transformation services that focus on delivering measurable ROI and quick results. Their goal is to enhance productivity and reduce operational costs for clients across various sectors, including retail and public health.

Naperville, IllinoisHeadquarters

2000Year Founded

$1.3MTotal Funding

PRE_SEEDCompany Stage

Consulting, Consumer Software, Enterprise SoftwareIndustries

1,001-5,000Employees

Benefits

Fun team outings

Health insurance

401(K)

Dental & vision insurance

Paid time-off

Food & drinks

Parental leave

Risks

Emerging tech startups offering similar solutions at lower costs threaten Egen's market share.

Rapid technological changes may outpace Egen's ability to update its solutions.

Economic downturns in retail and e-commerce could impact Egen's revenue.

Differentiation

Egen offers tailored digital transformation services with a focus on ROI and innovation.

The company specializes in secure, dynamic API strategies for seamless business growth.

Egen provides two application development paths: Rebuild and Replace, ensuring flexibility.

Upsides

Rising demand for cloud-native applications boosts Egen's cloud solutions offerings.

Growing interest in API-first development enhances Egen's API strategies and integration services.

Increased focus on cybersecurity in API development aligns with Egen's secure solutions.

Land your dream remote job 3x faster with AI

Try Jobo Free