Senior Data Engineer

Employment Type: Full time

Position Overview

Abbott is a global healthcare leader dedicated to helping people live more fully at all stages of life. Our innovative medical device businesses offer technologies that are faster, more effective, and less invasive, improving the lives of individuals with chronic conditions and diseases.

We are seeking a Senior Data Engineer to join our Global Data Science & Analytics organization. This role will be based in Sylmar, CA, or Santa Clara, CA. The Senior Data Engineer will be instrumental in designing, building, and maintaining robust data pipelines using Azure Databricks and PySpark. This position requires a strong understanding of data processing, ETL, SQL, and data governance, with a collaborative approach to working with data scientists, clinical scientists, and analysts.

Responsibilities

• Data Pipeline Development: Design, build, and maintain robust data pipelines using Azure Databricks and PySpark, ensuring optimal performance and scalability.
• ETL Implementation: Develop and optimize ETL pipelines utilizing Databricks tools to facilitate efficient data processing and transformation.
• Data Processing: Leverage Apache Spark for both batch and incremental data processing, ensuring data integrity and consistency.
• Collaboration: Work closely with data scientists and analysts to comprehend data requirements and deliver tailored solutions that meet business needs.
• SQL Development: Craft and refine complex SQL queries for effective data extraction, analysis, and reporting.
• R Utilization: Apply R for advanced data manipulation and statistical analysis, enhancing data insights (nice to have).
• Data Governance: Implement stringent data governance practices using tools such as Unity Catalog to ensure data quality, security, and compliance.
• Cloud Integration: Integrate Databricks with cloud platforms, with a preference for Azure, to streamline data workflows and enhance accessibility.
• Project Management: Manage multiple projects simultaneously, ensuring timely delivery and adherence to high-quality standards.
• Best Practices: Establish and enforce best practices for data quality, security, and governance across various projects.
• Issue Resolution: Troubleshoot and resolve data-related issues promptly to maintain operational efficiency.
• Continuous Learning: Stay abreast of the latest trends and advancements in data engineering to continuously improve processes and technologies.
• Stakeholder Communication: Communicate effectively with team members and stakeholders to ensure alignment and successful project outcomes.
• Advanced Databricks and Azure Integration: Utilize advanced Databricks functionalities and integrate with Azure services to optimize data pipelines, develop high-quality code, and implement automation for CI/CD.

Requirements

• Experience designing, building, and maintaining robust data pipelines using Azure Databricks and PySpark.
• Proficiency in developing and optimizing ETL pipelines.
• Experience leveraging Apache Spark for data processing.
• Ability to craft complex SQL queries.
• Experience implementing data governance practices.
• Familiarity with cloud platforms, preferably Azure.
• Strong project management skills for handling multiple concurrent projects.
• Excellent communication and problem-solving abilities.
• Knowledge of R for advanced data manipulation is a plus.

Company Information

Abbott is a global healthcare leader with leading businesses and products in diagnostics, medical devices, nutritionals, and branded generic medicines. Our 114,000 colleagues serve people in more than 160 countries, providing life-changing technologies across the healthcare spectrum.