Senior Data Engineer (Pipeline)
DemandbaseSalary not specified
Not SpecifiedCompensation
Senior (5 to 8 years)Experience Level
Full TimeJob Type
UnknownVisa
TechnologyIndustries
Requirements
- 5+ years of professional experience in software or data engineering using Python
- Strong understanding of software engineering best practices (testing, version control, CI/CD)
- Proven experience building and optimizing ETL/ELT pipelines and data workflows
- Proficiency in SQL and database concepts
- Experience with data processing frameworks (e.g., Pandas)
- Understanding of software design patterns and scalable architecture principles
- Experience with cloud platforms (GCP preferred)
- Knowledge of CI/CD pipelines and Infrastructure as Code tools
- Familiarity with containerization (Docker, Kubernetes)
- Bachelor’s degree in Computer Science, Engineering, or a related field (or equivalent experience)
- Excellent problem-solving, analytical, and communication skills
- Preferred Qualifications
- Experience with GCP services such as Cloud Run and Dataflow
- Experience with stream processing technologies (e.g., Pub/Sub)
- Familiarity with workflow orchestration tools (e.g., Airflow)
- Exposure to data visualization tools or libraries
- Knowledge of GitLab CI/CD and Terraform
- Experience with Snowflake, BigQuery, or Databricks
- GCP Data Engineer Certification
Responsibilities
- Design, develop, and maintain data pipelines and ETL/ELT processes using Python
- Build and optimize scalable, high-performance data applications
- Collaborate with cross-functional teams to define requirements and deliver reliable solutions
- Develop and manage real-time streaming pipelines using Pub/Sub or Apache Beam
- Participate in code reviews, architecture discussions, and continuous improvement initiatives
- Monitor, troubleshoot, and optimize production data systems for reliability and performance
Skills
Python
SQL
Pandas
ETL
ELT
Apache Beam
Pub/Sub
GCP
Docker
Kubernetes
CI/CD
ShyftLabs
Data-driven decision-making solutions for organizations
About ShyftLabs
ShyftLabs helps organizations adopt a data-first approach to their decision-making processes. Their services focus on establishing systems that enable companies to make quicker and more informed decisions based on data analysis. This approach allows businesses to gain insights that can keep them ahead of their competitors. Unlike other companies that may offer generic consulting services, ShyftLabs emphasizes the importance of data in driving decisions, ensuring that organizations can leverage their data effectively to enhance their strategic planning and operational efficiency.
None, CanadaHeadquarters
2018Year Founded
VENTURE_UNKNOWNCompany Stage
Data & Analytics, ConsultingIndustries
11-50Employees
Benefits
Health Insurance
Hybrid Work Options
Professional Development Budget
Risks
Increased competition from startups offering innovative, cost-effective solutions.
Growing demand for in-house analytics teams reducing reliance on consultants.
Rapid AI advancements may outpace ShyftLabs' current technology offerings.
Differentiation
ShyftLabs specializes in data governance, warehousing, and predictive analysis services.
The firm empowers organizations with a data-first approach for decision-making.
ShyftLabs establishes processes for faster, insightful decisions to outpace competition.
Upsides
Increased demand for data governance due to stricter privacy regulations.
Growing interest in predictive analytics in retail for inventory optimization.
Rising adoption of cloud-based BI tools among SMEs for cost-effectiveness.
Related Jobs
RemoteRemoteSenior Data Automation Professional
HumanaSalary not specified
RemoteStaff Data Engineer
Life360$166,500 - $245,000/year
Full Time
Expert & Leadership (9+ years)
RemoteStaff Integration Engineer
Chainalysis$110,000 - $150,000/year
- Full Time
- Senior (5 to 8 years)
RemoteSenior Cloud Engineer - Product Metrics
ClickhouseSalary not specified
- Full Time
- Senior (5 to 8 years)
RemoteAnalytics Engineer
Machinify$180,000 - $200,000/year
- Full Time
- Mid-level (3 to 4 years)
RemoteStaff Database Administrator
Wikimedia FoundationSalary not specified
- Full Time
- Mid-level (3 to 4 years)
RemoteData Platform Software Engineer
Muon SpaceSalary not specified
- Full Time
- Senior (5 to 8 years)