ShyftLabs

Staff Data Engineer

Toronto, Ontario, Canada

Not SpecifiedCompensation
Expert & Leadership (9+ years)Experience Level
Full TimeJob Type
UnknownVisa
Data Products, BiotechnologyIndustries

Staff Data Engineer

Position Overview

ShyftLabs is seeking an experienced Staff Data Engineer to implement and optimize enterprise data pipelines and ingestion frameworks for Fortune 500 companies. You will build robust Databricks-based data processing infrastructure, implement pipeline reliability solutions, and create scalable technical solutions for both batch and streaming data workloads.

ShyftLabs is a growing data product company founded in early 2020 and works primarily with Fortune 500 companies. We deliver digital solutions built to help accelerate the growth of businesses in various industries, by focusing on creating value through innovation.

Job Responsibilities

  • Implement and optimize enterprise-scale data pipeline architecture processing 1500+ daily jobs across multiple business domains.
  • Build and maintain Databricks job orchestration solutions integrated with Active Batch scheduling platforms.
  • Implement data ingestion frameworks supporting CDC via AWS DMS and S3-based data lake patterns.
  • Troubleshoot and resolve pipeline reliability issues including overrun problems, dependency failures, and compute optimization.
  • Build observability and monitoring solutions for large-scale data operations teams.
  • Implement near real-time ingestion patterns using DLT (Delta Live Tables) and streaming architectures.
  • Build and maintain data quality frameworks integrated with data governance processes.
  • Implement serverless migration strategies and optimize cloud resource utilization.
  • Lead technical workshops and provide hands-on guidance to client engineering teams.

Requirements

Enterprise Pipeline Expertise

  • Proven experience building and optimizing data pipelines at Fortune 500 scale.

Operational Excellence

  • Deep understanding of pipeline monitoring, alerting, and failure recovery patterns.

Performance Optimization

  • Ability to analyze and optimize compute resources and job execution patterns.

Technical Consulting

  • Strong workshop facilitation and client relationship skills, with ability to present technical solutions and lead hands-on training sessions with client engineering teams.

Basic Qualifications

  • Bachelor's or Master's degree in Computer Science, Engineering, or related technical field.
  • 6+ years of experience in enterprise data engineering with Fortune 500 companies.
  • Expert-level experience with Databricks platform including job scheduling and cluster management.
  • Hands-on experience with Active Batch or similar enterprise job orchestration platforms.
  • Deep knowledge of AWS data services including DMS, S3, and database connectivity.
  • Proven experience with CDC (Change Data Capture) implementations at enterprise scale.
  • Strong background in Delta Live Tables (DLT) and streaming data processing.
  • Understanding of Apache Spark optimization and performance tuning.
  • Experience with enterprise database integration and cross-platform data connectivity.

Preferred Qualifications

  • Experience with enterprise integration patterns and API data ingestion.
  • Knowledge of cloud storage optimization and data lake best practices.
  • Background in high-volume transaction processing systems.
  • Experience with retail or energy industry data patterns.
  • Familiarity with trading and pricing systems data workflows.

Company Information

ShyftLabs is a growing data product company founded in early 2020 and works primarily with Fortune 500 companies. We deliver digital solutions built to help accelerate the growth of businesses in various industries, by focusing on creating value through innovation.

We are proud to offer a competitive salary alongside a strong healthcare insurance and benefits package. The role is preferably hybrid, with 2 days per week spent in the office, and flexibility for client engagement needs. We pride ourselves on the growth of our employees, offering extensive learning and development resources.

ShyftLabs is an equal-opportunity employer committed to creating a safe, diverse and inclusive environment. We encourage qualified applicants of all backgrounds including ethnicity, religion, disability status, gender identity, sexual orientation, family status, age, nationality, and education levels to apply. If you are contacted for an interview and require accommodation during the interviewing process, please let us know.

Employment Type

  • Full-Time

Location Type

  • [Information not specified]

Salary

  • [Information not specified]

Skills

Data Pipelines
Data Ingestion
Databricks
AWS DMS
S3
Data Lake
CDC
DLT
Streaming Architectures
Data Quality
Data Governance
Serverless
Cloud Resource Optimization
Batch Processing
Active Batch
Observability
Monitoring
Failure Recovery
Compute Optimization

ShyftLabs

Data-driven decision-making solutions for organizations

About ShyftLabs

ShyftLabs helps organizations adopt a data-first approach to their decision-making processes. Their services focus on establishing systems that enable companies to make quicker and more informed decisions based on data analysis. This approach allows businesses to gain insights that can keep them ahead of their competitors. Unlike other companies that may offer generic consulting services, ShyftLabs emphasizes the importance of data in driving decisions, ensuring that organizations can leverage their data effectively to enhance their strategic planning and operational efficiency.

None, CanadaHeadquarters
2018Year Founded
VENTURE_UNKNOWNCompany Stage
Data & Analytics, ConsultingIndustries
11-50Employees

Benefits

Health Insurance
Hybrid Work Options
Professional Development Budget

Risks

Increased competition from startups offering innovative, cost-effective solutions.
Growing demand for in-house analytics teams reducing reliance on consultants.
Rapid AI advancements may outpace ShyftLabs' current technology offerings.

Differentiation

ShyftLabs specializes in data governance, warehousing, and predictive analysis services.
The firm empowers organizations with a data-first approach for decision-making.
ShyftLabs establishes processes for faster, insightful decisions to outpace competition.

Upsides

Increased demand for data governance due to stricter privacy regulations.
Growing interest in predictive analytics in retail for inventory optimization.
Rising adoption of cloud-based BI tools among SMEs for cost-effectiveness.

Land your dream remote job 3x faster with AI