Senior Data Engineer ( Azure ) at Egen

Hyderabad, Telangana, India

Apply Now

Not SpecifiedCompensation

Senior (5 to 8 years)Experience Level

Full TimeJob Type

UnknownVisa

N/AIndustries

Requirements

Total experience: 8 to 10 years
Deep expertise in Microsoft Azure data ecosystem (Azure Data Factory, Azure Databricks, Synapse Analytics, Azure Data Lake Storage Gen2, Azure Functions/Logic Apps, Azure Event Hubs/Kafka/Stream Analytics)
Strong proficiency in Python or Scala for data transformations
Proficient in SQL (T-SQL, Spark SQL, or Synapse SQL)
Experience with PySpark for distributed data processing
Knowledge of Git, CI/CD, and DevOps best practices
Expertise in data warehouse design, data lakehouse architecture, data pipelines, data modeling (star schema, snowflake, data vault, dimensional modeling, slowly changing dimensions, fact tables), data partitioning, indexing, and query optimization
Experience with Infrastructure as Code (IaC) tools such as Terraform, Bicep, or ARM templates
Familiarity with data governance (metadata management, lineage tracking, data cataloging e.g., Azure Purview), security/compliance (Azure Active Directory, RBAC), and automation (DevOps practices)
Experience managing data ingestion from diverse sources (APIs, on-prem databases, SaaS applications, data lakes) for batch and streaming use cases
Familiarity with Power BI integration or other reporting tools (plus)
Experience with API-based data ingestion and RESTful integrations (plus)
Exposure to machine learning data pipelines or feature stores (plus)
Excellent analytical and problem-solving abilities
Strong communication and documentation skills
Ability to work in an agile, fast-paced environment
Proven experience collaborating in cross-functional teams
Self-driven with passion for data engineering and cloud technologies

Responsibilities

Design, develop, and maintain data ingestion, transformation, and storage pipelines using Azure Data Factory, Azure Databricks, and Synapse Analytics
Build end-to-end ETL/ELT workflows from diverse sources such as APIs, on-prem databases, SaaS applications, and data lakes
Implement data modeling (star schema, snowflake, data vault, etc.) for analytical and operational data stores
Manage data ingestion frameworks for both batch and streaming (real-time) use cases using Azure Event Hubs / Azure Stream Analytics / Kafka
Ensure data quality, consistency, and integrity across all environments
Implement and enforce data governance standards, including metadata management, lineage tracking, and data cataloging (e.g., Azure Purview)
Manage security, compliance, and access controls using Azure Active Directory (AAD) and RBAC principles
Automate data validation, auditing, and monitoring workflows using modern DevOps practices
Manage and optimize Azure Data Lake Storage (ADLS Gen2) for cost and performance
Leverage Infrastructure as Code (IaC) tools such as Terraform, Bicep, or ARM templates for environment provisioning
Implement CI/CD pipelines for data workflows using Azure DevOps / GitHub Actions
Optimize compute and storage costs while maintaining scalability and resilience
Partner with data scientists, BI developers, and product teams to deliver reliable data solutions
Mentor junior data engineers on best practices, coding standards, and Azure services
Participate in architectural reviews and contribute to data platform design decisions
Communicate technical insights and trade-offs to non-technical stakeholders effectively

Skills

Key technologies and capabilities for this role

Azure Data FactoryAzure DatabricksSynapse AnalyticsAzure Event HubsAzure Stream AnalyticsKafkaAzure PurviewAzure Active DirectoryAzure Data Lake StorageTerraformBicepARM templatesAzure DevOpsGitHub Actions

Questions & Answers

Common questions about this position

What is the total experience required for this Senior Data Engineer role?

The position requires 8 to 10 years of total experience.

What are the core Azure services required for this position?

Core Azure services include Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage (ADLS Gen2), Azure Functions/Logic Apps, and Azure Event Hubs/Kafka/Stream Analytics.

What programming skills are needed for this role?

Strong proficiency in Python or Scala for data transformations and proficient in SQL (T-SQL) are required.

Is this a remote position or does it require office work?

This information is not specified in the job description.

What does the team collaboration look like for this role?

The role involves working with cross-functional teams including data scientists, analysts, and business stakeholders, partnering with data scientists, BI developers, and product teams, and mentoring junior data engineers.

Egen

Provides API strategies and cloud solutions

About Egen

Egen Solutions specializes in API strategies and cloud solutions to help businesses modernize and scale. They offer two main paths for application development: Rebuild, which improves existing functionalities, and Replace, which creates new applications in the cloud. Egen stands out by providing tailored digital transformation services that focus on delivering measurable ROI and quick results. Their goal is to enhance productivity and reduce operational costs for clients across various sectors, including retail and public health.

Naperville, IllinoisHeadquarters

2000Year Founded

$1.3MTotal Funding

PRE_SEEDCompany Stage

Consulting, Consumer Software, Enterprise SoftwareIndustries

1,001-5,000Employees

Benefits

Fun team outings

Health insurance

401(K)

Dental & vision insurance

Paid time-off

Food & drinks

Parental leave

Risks

Emerging tech startups offering similar solutions at lower costs threaten Egen's market share.

Rapid technological changes may outpace Egen's ability to update its solutions.

Economic downturns in retail and e-commerce could impact Egen's revenue.

Differentiation

Egen offers tailored digital transformation services with a focus on ROI and innovation.

The company specializes in secure, dynamic API strategies for seamless business growth.

Egen provides two application development paths: Rebuild and Replace, ensuring flexibility.

Upsides

Rising demand for cloud-native applications boosts Egen's cloud solutions offerings.

Growing interest in API-first development enhances Egen's API strategies and integration services.

Increased focus on cybersecurity in API development aligns with Egen's secure solutions.

Land your dream remote job 3x faster with AI

Try Jobo Free