RemoteLeaf Logo
Log in Sign up

Multilingual AI Quality Specialist

4 weeks, 1 day ago
Sweden, United Kingdom
Full-time
Mid Level
AI (Artificial Intelligence)
Data Science and Analytics
Generative AI LLM Python SQL
Apply Now
Spotify

Spotify

Spotify is a leading global audio streaming service empowering artists and inspiring fans worldwide with over 70 million tracks. It has 365 million users, including 165 million subscribers, across 178 markets.

Media
Founded 2006
View All Jobs 26

Description

  • Define quality frameworks, evaluation rubrics, thresholds, and methodologies for multilingual AI experiences.
  • Design and execute structured evaluations for AI-generated, AI-translated, AI-curated, and recommendation-driven experiences.
  • Lead multilingual dataset curation, annotation, enrichment, and ground-truth creation to support AI model development and evaluation.
  • Analyze evaluation results, identify quality gaps, and provide actionable recommendations to improve multilingual AI quality.
  • Support LLM-as-a-judge workflows, evaluator calibration, and human-AI agreement studies.
  • Partner with Product, Engineering, Data Science, Research, Localization, vendors, and market experts to improve AI quality signals and inform launch decisions.
  • Document best practices and help define quality standards across languages, markets, and AI use cases.
  • Contribute to building scalable evaluation capabilities for future AI-powered experiences across Spotify.

Requirements

  • Experience in multilingual quality evaluation, localization, data curation, annotation, AI evaluation, or related fields, including text-to-text and text-to-speech experiences.
  • Understanding of language quality, cultural relevance, content quality, and user experience across multiple languages and markets.
  • Experience designing or conducting structured evaluations using quality rubrics, audits, annotation projects, or review methodologies.
  • Comfort using qualitative and quantitative data to identify trends, measure quality, and make recommendations.
  • Familiarity with large language models (LLMs), generative AI evaluation, human-in-the-loop workflows, or LLM-as-a-judge methodologies.
  • Ability to work through ambiguity and turn complex quality challenges into practical evaluation strategies.
  • Strong communication skills and experience collaborating in highly cross-functional environments with technical and non-technical partners.
  • Experience with recommendation systems, personalization, search, ranking, machine translation, generative AI, dataset creation, annotation operations, evaluator calibration, prompt testing, model evaluation, SQL, Python, dashboards, or annotation platforms is a plus.
  • Role is based in London or Stockholm.
  • Ability to work with some in-person meetings while maintaining flexibility to work from home.

Benefits

  • Flexible work environment with the ability to work from home.
  • Some in-person meetings with flexibility to work where you work best.
  • Reasonable accommodations available during the interview process and application stages.
  • Equal opportunity employer with an inclusive workplace culture.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Prolific

AI Trainer - Advanced Mandarin Fluency - United Kingdom

Prolific 51-250 Professional Services

Prolific is hiring fluent Mandarin speakers for a remote, task-based role that uses live video conversations to help train AI models on natural human speech.

United Kingdom Contract Entry Level AI (Artificial Intelligence)
$37k-$37k
7 hours, 58 minutes ago
Prolific

AI Training - Fluent Russian Speakers - Cyprus

Prolific 51-250 Professional Services

Prolific is hiring a remote AI Trainer for Russian to conduct live, task-based video conversations that help improve AI models’ understanding of natural human speech.

Cyprus Contract Entry Level AI (Artificial Intelligence)
$37k-$37k
7 hours, 58 minutes ago
Prolific

AI Trainer - Advanced Mandarin Fluency - London

Prolific 51-250 Professional Services

Prolific is hiring fluent Mandarin speakers for a remote, task-based AI training role that uses live video conversations to help improve how AI models understand natural human speech.

Anywhere Freelance Entry Level AI (Artificial Intelligence)
$37k-$37k
7 hours, 58 minutes ago
Prolific

AI Trainer - Advanced Mandarin Fluency - Singapore

Prolific 51-250 Professional Services

Prolific is hiring fluent Mandarin speakers for remote, task-based participant work to join live conversations that help train AI models on natural human dialogue.

Singapore Freelance Entry Level AI (Artificial Intelligence)
$37k-$37k
7 hours, 58 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers