Senior Data Engineer

Position Summary

We are seeking a Senior Data Engineer with specialized expertise in data streaming technologies to join our data team. This role focuses on building and maintaining high-performance data streaming architectures that enable real-time data processing and analytics. The ideal candidate will have deep experience with Apache Kafka, AWS Managed Streaming for Apache Kafka (MSK), Amazon Kinesis, and related streaming technologies in cloud environments.

Role Focus

A Senior Data Engineer at Effectual is primarily responsible for building and maintaining the streaming data architecture that enables real-time data processing and analytics. This involves constructing robust data streaming pipelines that transform and transport data from various sources in real-time, ensuring data flows efficiently through streaming systems for immediate analysis and operational decision-making. You will focus on the efficient and secure management of streaming data systems, ensuring that data is processed, stored, and made available for real-time analytics and downstream applications.

Essential Duties and Responsibilities

Streaming Data Architecture & Pipeline Development

• Design, build, and maintain scalable streaming data architectures using Kafka, MSK, and Kinesis
• Develop real-time data pipelines that handle high-volume, high-velocity data streams
• Implement event-driven architectures and microservices patterns for streaming data processing
• Create and optimize data streaming topologies for complex event processing scenarios
• Design fault-tolerant streaming systems with proper error handling and data recovery mechanisms

Kafka & MSK Management

• Configure, deploy, and manage Apache Kafka clusters and AWS MSK environments
• Implement Kafka Connect pipelines for streaming data integration
• Design optimal Kafka topic partitioning strategies and replication configurations
• Monitor and optimize Kafka cluster performance, throughput, and latency
• Implement Kafka security configurations including SSL/TLS, SASL, and ACLs
• Manage Kafka Schema Registry for data serialization and evolution

Kinesis & AWS Streaming Services

• Design and implement Amazon Kinesis Data Streams and Kinesis Data Firehose solutions
• Configure Kinesis Analytics applications for real-time stream processing
• Optimize Kinesis shard management and auto-scaling configurations
• Implement Kinesis data retention and archival strategies
• Integrate Kinesis with other AWS services for comprehensive streaming solutions

Data Processing & Analytics

• Develop real-time stream processing applications using Apache Spark Streaming, Kafka Streams, or AWS Lambda
• Implement complex event processing (CEP) patterns for real-time analytics
• Build streaming ETL pipelines that transform data in motion
• Create real-time aggregations, windowing operations, and stateful stream processing
• Optimize streaming query performance and resource utilization

Integration & Data Flow Management

• Ensure seamless integration between streaming systems and data lakes, data warehouses, and operational databases
• Implement data lineage and monitoring for streaming data pipelines
• Create automated data quality checks and validation for streaming data
• Manage data serialization formats (Avro, JSON, Protobuf) and schema evolution
• Coordinate with data scientists and analysts to ensure streaming data meets analytical requirements

DevOps & Infrastructure Management

• Implement Infrastructure as Code (IaC) for streaming data platforms using Terraform or CloudFormation
• Automate deployment and management of streaming infrastructure through CI/CD pipelines
• Monitor streaming system health, performance metrics, and alerting
• Implement disaster