Senior Site Reliability Engineer (SRE)

Employment Type: [Not specified] Location Type: [Not specified] Salary: [Not specified]

Position Overview

The Wikimedia Foundation is seeking a Senior Site Reliability Engineer (SRE) to join our Service Operations SRE team. This team is responsible for the infrastructure that powers wikipedia.org and other Wikimedia Foundation projects. Our SRE team is a distributed, diverse group of engineers passionate about exploring, experimenting with, and adopting new technologies. The Service Operations team specifically focuses on the application layer of our complex infrastructure and our developer-facing services.

Our technology stack includes MediaWiki instances running on bare metal, surrounding microservices on our in-house Kubernetes platform, a custom-designed deployment pipeline, and open-source collaboration software such as Gerrit, Gitlab, and Phabricator. We are strong advocates for automation, utilizing Python, Puppet, and Spicerack (https://github.com/wikimedia/operations-software-spicerack) for this purpose. We operate transparently, documenting our work on our internal task management system and our technical wiki (https://wikitech.wikimedia.org). We encourage you to explore our activities there!

As a senior engineer on the SRE Service Operations team, you will play a key role in designing and managing the infrastructure and services that underpin the Wikimedia Foundation’s projects. This includes, but is not limited to, Kubernetes clusters, application servers, associated datastores, and other developer-facing services. You will be an integral part of our 24/7 incident response process and participate in the on-call rotation. This role involves close collaboration with other SRE team members to maintain and enhance our systems, as well as interaction with non-SRE teams such as Security, Release, and Software Engineering, all working together to advance our projects and technologies.

We utilize a broad range of technologies, and we value your experience even if it doesn't perfectly align with every keyword listed. If our work interests you and you are drawn to the prospect of working with a globally distributed team that manages a Top-15 website, we might be the right fit for you.

Responsibilities

• Design, implement, and maintain public-facing infrastructure and services.
• Utilize configuration management and deployment tools.
• Engage in architectural design and operation at scale.
• Monitor systems and services, optimizing performance and resource utilization.
• Proactively identify sources of instability in distributed systems and analyze complex system failures from a reliability and resilience perspective.
• Perform common operating system tasks, including logging and backup/restore operations.
• Implement cookbooks/runbooks for routine maintenance actions.
• Participate in a 24/7 on-call rotation and escalations for production issue resolution.
• Lead incident response efforts and post-incident reviews, contributing to failure analysis and implementing preventive measures.
• Automate and streamline tasks, identifying and addressing process gaps.
• Collaborate effectively within a global, asynchronously communicating team (support will be provided for adjusting to remote work if needed).
• Mentor peers in areas of technical and operational expertise.
• Be expected to travel domestically or potentially internationally 2-3 times per year for team gatherings and conferences.

Skills and Experience

• 5+ years of experience in an SRE/Operations/DevOps role.
• Proven experience operating highly available infrastructure.
• Experience running applications and services at scale.
• Experience implementing containerization solutions (e.g., Docker, Kubernetes).
• Proficiency in shell scripting and at least one programming language commonly used in SRE/Operations engineering (e.g., Python, Go, Ruby).
• Comfort with open-source configuration management and orchestration tools (e.g., Puppet, Ansible, Terraform).
• Strong technical English communication skills.

Additionally, we’d love it if you have:

• Experience with package management for operating systems (e.g., Debian).
• We are avid supporters (and users) of o[...content truncated...]