Job Description: Expert Rater

Employment Type: Full-time

---

Position Overview

Welo Data is an award-winning localization and AI data company that runs some of the world’s largest AI model relevance rating programs. This role will play a crucial role in analyzing and improving AI-driven online advertising platforms. You will help establish benchmarks and guidelines for data used to train AI models, particularly in areas lacking defined standards, thus shaping the future of web-based advertising and AI models.

---

Responsibilities

• Rate Client-Provided Ads: Evaluate advertisements based on specific quality, relevance, and compliance criteria provided by the client, ensuring ratings are objective, consistent, and aligned with predefined standards.
• Guideline Management:
  • Identify guideline ambiguities and apply critical thinking when assessing borderline cases.
  • Review and refine rating guidelines, identifying areas for improvement and recommending updates to enhance accuracy, consistency, and overall rating quality.
• Dataset Curation: Curate high-quality datasets ("golden sets") by selecting and validating top-tier rated ads to serve as benchmarks for AI model training and human rater calibration.
• Collaboration: Collaborate with cross-functional teams (e.g., quality assurance, data analysts, and AI training teams) to ensure rating processes align with evolving industry and client needs.
• Analysis and Insights: Analyze trends and patterns in ad ratings, providing insights that contribute to guideline optimizations and overall rating effectiveness.
• Mentorship: Assist in training and mentoring new raters, ensuring they understand guidelines and best practices for accurate evaluations.
• Efficiency: Maintain a high level of accuracy and efficiency, meeting daily/weekly rating quotas.

---

Requirements

• Experience:
  • Experience in AI model relevance, such as data labeling, linguistic annotation, or relevance rating.
  • Proven track record of data-based decision making; analytical and data-driven.
  • Strong interest and understanding of digital marketing and advertising, especially as it relates to artificial intelligence.
  • Experience researching and gathering insights, and understanding complex problems across a variety of topics.
  • Ability to effectively summarize complex information, questions, and solutions for various audiences.
  • Ability to define, explain, and defend points of view in group discussions.
  • Comfortable reviewing and discussing a variety of topics; willingness to research and understand to support consensus decisions is vital.
  • Comfortable with ambiguity in guidelines, workflows, and processes, and motivated to help establish those systems.
• Language Skills:
  • Native-level fluency in the target language.
  • Excellent understanding of both written and spoken English.
• Education/Work Experience:
  • A 4-year university degree and/or at least 2 years of work experience in one or more of the following: AI-based relevance rating, marketing/advertising, linguistics, content creation, writing, tech-related quality assurance, data labeling, or linguistic annotation.

---

Application Instructions

If you have a Candidate Login already, but have forgotten your password, please use the steps to reset your password. If you have forgotten your email login, please contact servicedesk@welocalize.com with the subject "Workday Candidate Login".

NOTICE: When creating your Workday account and entering personal information like name and address, please do not use ALL CAPS.

For EMEA Jobs, please review the Privacy Policy here.

---

Company Information

Welo Data is an award-winning localization and AI data company.