Senior Site Reliability Engineer (SRE)

Employment Type:

Location Type:

Salary:

Position Overview

The Wikimedia Foundation is seeking a Senior Site Reliability Engineer (SRE) to join our Service Operations SRE team. This team is responsible for the infrastructure that powers wikipedia.org and other Wikimedia Foundation projects. We are a distributed, diverse team passionate about exploring, experimenting with, and embracing new technologies. Our focus is on the application layer of our complex infrastructure and our developer-facing services.

Our technology stack includes MediaWiki instances on bare metal, microservices on our in-house Kubernetes platform, a custom deployment pipeline, and open-source collaboration software (Gerrit, Gitlab, Phabricator). We are strong proponents of automation, utilizing Python, Puppet, and Spicerack (https://github.com/wikimedia/operations-software-spicerack) for this purpose. We operate transparently, documenting our work on our internal task management system and our technical wiki (https://wikitech.wikimedia.org).

As a Senior SRE, you will be instrumental in designing, running, and maintaining the infrastructure and services that underpin the Wikimedia Foundation's projects. This includes Kubernetes clusters, application servers, associated datastores, and developer-facing services. You will be an active participant in our 24/7 incident response and on-call rotation. This role involves close collaboration with fellow SREs to enhance our systems and interaction with other engineering teams, including Security, Release, and Software Engineers, to drive technological advancements.

We utilize a broad range of technologies, and we encourage you to apply even if your experience doesn't precisely match every keyword listed. If our mission and the opportunity to work with a globally distributed team managing a Top-15 website appeal to you, we'd be interested in hearing from you.

Responsibilities

• System Design and Maintenance: Design, implement, and maintain public-facing infrastructure and services.
• Tooling and Automation: Utilize configuration management and deployment tools; implement cookbook/runbook procedures for common maintenance tasks.
• Scalability and Operations: Engage in architectural design and operation of systems at scale.
• Monitoring and Optimization: Monitor systems and services, optimizing performance and resource utilization.
• Reliability and Resilience: Proactively identify sources of instability in distributed systems and analyze complex system failure modes from a reliability and resilience perspective.
• System Administration: Perform common operating system tasks, including logging and backup/restore procedures.
• Incident Management: Participate in a 24/7 on-call rotation and escalations for production issue resolution. Lead incident response efforts, conduct post-incident reviews, contribute to failure analysis, and implement preventive measures.
• Process Improvement: Automate and streamline tasks, identifying and addressing process gaps.
• Collaboration: Collaborate effectively within a global, asynchronously communicating team. We provide support for adapting to remote work.
• Mentorship: Mentor peers in your areas of technical and operational expertise.
• Travel: Expected to travel domestically or potentially internationally 2-3 times per year for team gatherings and conferences.

Skills and Experience

Required

• 5+ years of experience in an SRE/Operations/DevOps role.
• Experience operating highly available infrastructure.
• Experience running applications and services at scale.
• Experience implementing containerization solutions (e.g., Docker, Kubernetes).
• Proficiency in shell scripting and at least one programming language commonly used in SRE/Operations engineering (e.g., Python, Go, Ruby).
• Comfort with open-source configuration management and orchestration tools (e.g., Puppet, Ansible, Terraform).
• Communicative technical English proficiency.

Preferred

• Experience with operating system package management (e.g., Debian).
• We are avid supporters (and users) of o... (Content truncated)