Senior Site Reliability Engineer (SRE)

Employment Type:

Location Type:

Salary:

Position Overview

The Wikimedia Foundation is seeking a Senior Site Reliability Engineer (SRE) to join our Service Operations SRE team. This team is responsible for the infrastructure that powers wikipedia.org and other Wikimedia Foundation projects. The SRE team is a distributed, diverse group of engineers passionate about exploring, experimenting with, and adopting new technologies. Our Service Operations team focuses on the application layer of our complex infrastructure and developer-facing services.

Our technology stack includes MediaWiki instances on bare metal, microservices on our in-house Kubernetes platform, a custom deployment pipeline, and open-source collaboration software (Gerrit, Gitlab, Phabricator). We are strong advocates for automation, primarily using Python, Puppet, and Spicerack (https://github.com/wikimedia/operations-software-spicerack). We operate transparently, documenting our work on our internal task management system and technical wiki (https://wikitech.wikimedia.org).

As a Senior SRE in Service Operations, you will be instrumental in designing and managing the infrastructure and services that underpin the Wikimedia Foundation's projects. This includes, but is not limited to, Kubernetes clusters, application servers, related datastores, and developer-facing services. You will actively participate in 24/7 incident response and on-call rotations. This role involves close collaboration with other SRE team members to maintain and enhance our systems, as well as interaction with non-SRE teams such as Security, Release, and Software Engineering, all working together to advance our projects and technologies.

We utilize a wide array of technologies, and we encourage you to apply even if your experience doesn't perfectly align with every keyword listed. If our mission and work resonate with you, and you enjoy the prospect of working with a globally distributed team managing a Top-15 website, we might be the perfect fit.

Responsibilities

• Design, implement, and maintain public-facing infrastructure and services.
• Utilize configuration management and deployment tools.
• Engage in architectural design and operation at scale.
• Monitor systems and services, optimizing performance and resource utilization.
• Proactively identify sources of instability in distributed systems and analyze complex system failures from a reliability and resilience perspective.
• Perform common operating system tasks, including logging and backup/restore operations.
• Implement cookbooks/runbooks for routine maintenance actions.
• Participate in a 24/7 on-call rotation and escalations for production issue resolution.
• Lead incident response efforts and post-incident reviews, contributing to failure analysis and implementing preventive measures.
• Automate and streamline tasks, identifying and addressing process gaps.
• Collaborate effectively within a global, asynchronously communicating team (support will be provided for remote work adaptation).
• Mentor peers in areas of technical and operational expertise.
• Expected to travel domestically or potentially internationally 2-3 times per year for team gatherings and conferences.

Skills and Experience

• 5+ years of experience in an SRE/Operations/DevOps role.
• Proven experience operating highly available infrastructure.
• Experience running applications and services at scale.
• Experience implementing containerization solutions (e.g., Docker, Kubernetes).
• Proficiency in shell scripting and at least one programming language commonly used in SRE/Operations engineering (e.g., Python, Go, Ruby).
• Comfort with open-source configuration management and orchestration tools (e.g., Puppet, Ansible, Terraform).
• Communicative technical English proficiency.

Additionally, we’d love it if you have:

• Experience with package management for operating systems (e.g., Debian).
• We are avid supporters (and users) of o...