Sr. Data Engineer/Tech Lead at Eli Lilly and Company

Bengaluru, Karnataka, India

Apply Now

Not SpecifiedCompensation

Senior (5 to 8 years), Expert & Leadership (9+ years)Experience Level

Full TimeJob Type

UnknownVisa

Healthcare, PharmaceuticalsIndustries

Requirements

10+ years of professional experience in data engineering or related roles
Expert-level proficiency in Python for data engineering, including data processing libraries (Pandas, PySpark, Dask, Polars), API development (FastAPI, Flask), and testing (Pytest, unittest)
Strong AWS expertise with hands-on experience in: Data Storage (S3, RDS/Aurora, DynamoDB, Redshift), Data Processing (Glue ETL jobs, crawlers, Data Catalog; EMR, Athena), Streaming (Kinesis Data Streams, Firehose, Analytics; MSK Managed Kafka), Orchestration (Step Functions, EventBridge, Lambda), Analytics (QuickSight, Athena, Redshift Spectrum), Data Lake (Lake Formation, Glue Data Catalog), Infrastructure (CloudFormation, CDK, IAM, VPC, CloudWatch)
Workflow Orchestration: Apache Airflow (strong preference)
Big Data Technologies: Apache Spark (PySpark) for distributed data processing
Expert skills in ETL/ELT, data integration, ML Ops, and SQL
Intermediate to advanced skills in Python, Pyspark, AI/ML, and data visualization
Ability to review, optimize, document, and mentor data/visualization engineers on data pipelines, mapping, cleansing, and visual design using various tools and platforms
Ability to break down moderately complex problems to implement for increased business impact
Ability to support other team members, actively share learnings, drive and enforce team process improvements, and promote new and innovative ideas across multiple teams

Responsibilities

Hands-On Development (75%): Build and maintain scalable data platforms and infrastructure on AWS
Implement end-to-end data pipelines for batch and real-time data processing
Build robust ETL/ELT workflows to ingest, transform, and load data from diverse sources
Implement data lake/Lakehouse architectures using AWS S3, Glue, Athena, and Lake Formation
Design and optimize data warehouse solutions (Redshift, Snowflake) for analytics and reporting
Establish data quality frameworks and automated monitoring systems
Write production-quality Python code for data processing, transformation, and automation
Build scalable data pipelines using Apache Airflow, AWS Step Functions, or similar orchestration tools
Develop streaming data solutions using Kinesis, Kafka, or AWS MSK
Optimize SQL queries and database performance for large-scale datasets
Implement data validation, cleansing, and quality checks
Build APIs and microservices for data access and integration
Create monitoring, alerting, and observability solutions for data pipelines
Debug and resolve data pipeline failures and performance bottlenecks
Technical Leadership & Collaboration (25%): Mentor junior and mid-level data engineers through code reviews and technical guidance
Establish best practices for data engineering, testing, and deployment
Collaborate with data scientists, analysts, and business stakeholders to understand data requirements
Work with ML engineers to build data pipelines supporting machine learning workflows
Partner with platform/infrastructure teams on cloud architecture and cost optimization
Lead technical design discussions and architectural reviews
Document data architectures, pipelines, and processes
Evangelize data engineering best practices across the organization

Skills

Key technologies and capabilities for this role

ETLELTSQLPythonPysparkAWSS3GlueAthenaLake FormationRedshiftSnowflakeML OpsAI/MLdata visualizationdata pipelines

Questions & Answers

Common questions about this position

What technical skills are required for the Senior Data Engineer role?

The role requires expert skills in ETL/ELT, data integration, ML Ops, and SQL, as well as intermediate to advanced skills in Python, Pyspark, AI/ML, and data visualization.

What are the key responsibilities in hands-on development for this position?

Responsibilities include building scalable data platforms on AWS, implementing ETL/ELT workflows, data lake architectures using AWS S3, Glue, Athena, and Lake Formation, and optimizing data warehouses like Redshift and Snowflake.

Is this a remote position or does it require working from a specific location?

This information is not specified in the job description.

What is the salary or compensation for this role?

This information is not specified in the job description.

What leadership and collaboration duties does the role involve?

The role involves 25% technical leadership, including mentoring junior engineers, establishing best practices, collaborating with data scientists and stakeholders, and leading technical design discussions.

Eli Lilly and Company

Develops and delivers prescription medicines globally

About Eli Lilly and Company

Eli Lilly and Company is a global pharmaceutical company that focuses on discovering, developing, and delivering medicines to improve health. The company has a long history of scientific achievements, including the creation of insulin, the first life-saving treatment for diabetes. Lilly's operations involve extensive research and development to create new medications and enhance existing ones, ensuring they are safe and effective. Their products are primarily prescription medicines sold to healthcare providers for various medical conditions, including diabetes, cancer, and pain management. What sets Lilly apart from its competitors is its strong commitment to ethical practices and the protection of its products from counterfeiting. The company's goal is to enhance lives through innovative medical solutions while maintaining high standards of quality and ethics.

Indianapolis, IndianaHeadquarters

1876Year Founded

$1,180.1MTotal Funding

IPOCompany Stage

Biotechnology, HealthcareIndustries

10,001+Employees

Risks

Competition from Novo Nordisk's Ozempic may impact tirzepatide's market share.

Potential construction delays in Indiana could affect GLP-1 drug production timelines.

Regulatory challenges may hinder Kisunla's expansion in new Alzheimer's markets.

Differentiation

Eli Lilly's rich history includes the first life-saving insulin treatment.

Lilly's strategic partnerships enhance its position in neurodegenerative disease treatments.

FDA approval of Zepbound opens new therapeutic markets for sleep disorder treatments.

Upsides

Lilly's $9 billion complex in Indiana boosts GLP-1 drug production capacity.

Kisunla's approval in China expands Lilly's Alzheimer's treatment market in Asia.

Collaboration with EVA Pharma enhances Lilly's reputation as socially responsible.

Land your dream remote job 3x faster with AI

Try Jobo Free