Senior Data Engineer
FlexSalary not specified
- Full Time
- Senior (5 to 8 years)
Not SpecifiedCompensation
Senior (5 to 8 years)Experience Level
Full TimeJob Type
UnknownVisa
Data & AnalyticsIndustries
Position Overview
- Location Type: Remote
- Employment Type: Full-time
- Salary: Not specified
- Description: We are seeking an experienced Data Engineer to join our team. This role will focus on designing and enhancing data pipelines, maintaining data integrity, and supporting analytics projects.
Requirements
- Experience: 5 to 7+ years of relevant experience.
- Big Data Technologies: Extensive knowledge of big data technologies.
- ETL/ELT Workflows: Robust background in ETL/ELT processes.
- Data Modeling: Strong understanding of data modeling techniques.
- Mandatory Skills:
- PySpark: Proficient in using PySpark for data processing and ETL workflows.
- Azure Databricks: Experience with the Databricks platform, including cluster setup and management.
- Data Streaming: Knowledge of streaming data processing with frameworks such as Spark Streaming.
- Python: Strong programming skills in Python for scripting and automation tasks.
- SQL: Advanced skills in SQL for querying and managing relational databases.
- Spark Optimization: Experience in optimizing Spark applications for enhanced performance.
Responsibilities
- Design, Develop, and Maintain ETL/ELT Pipelines: Develop scalable ETL/ELT pipelines using PySpark and Databricks.
- Data Streaming Solutions: Implement and enhance data streaming solutions for real-time data processing.
- Spark Job Performance: Improve Spark job performance by addressing memory management, partitioning strategies, and implementing efficient data storage formats.
- Collaboration: Collaborate with data scientists and analysts to gather data requirements and provide reliable datasets for analysis.
- SQL Queries: Create and refine complex SQL queries for data extraction, transformation, and analysis.
- Data Quality & Integrity: Maintain data quality and integrity through automated testing and validation methods.
- Documentation & Metadata: Document data workflows and maintain metadata for governance purposes.
- Technology Research & Adoption: Research and adopt new data engineering technologies and methods to enhance efficiency and scalability.
Benefits
- Flexible Working Hours: Self-managed working hours.
- Competitive Salary: Not specified.
- Annual Bonus: Subject to company performance.
- Training & Development: Access to Udemy online training and opportunities for learning and growth.
Company Information
- Company: Mindera (Uses technology to build products we are proud of)
Skills
PySpark
Azure Databricks
Data Streaming
Python
SQL
Spark Optimization
Mineral
Develops AI tools for sustainable agriculture
About Mineral
Mineral.ai develops technology solutions aimed at improving the agriculture industry. The company utilizes perception technology, artificial intelligence (AI), and machine learning (ML) to create tools that help farmers, researchers, and agricultural advisors increase crop yields, manage pests, and adapt to climate change. Their products include precision agriculture tools that optimize resource use and advanced data analytics platforms that provide insights from agricultural data. Unlike many competitors, Mineral.ai focuses on creating partnerships within the agriculture sector to co-develop solutions, enhancing their product offerings. The goal of Mineral.ai is to support sustainable food production and help feed the world more efficiently.
Mountain View, CaliforniaHeadquarters
N/AYear Founded
VENTURE_UNKNOWNCompany Stage
Food & Agriculture, AI & Machine LearningIndustries
51-200Employees