Spotify

Spotify is a leading global audio streaming service empowering artists and inspiring fans worldwide with over 70 million tracks. It has 365 million users, including 165 million subscribers, across 178 markets.

Media

Consumer Discretionary

Founded 2006

26 open positions

Links

View All Jobs

Multilingual AI Quality Specialist

1 month ago

Sweden, United Kingdom

Full-time

Mid Level

AI (Artificial Intelligence)

Data Science and Analytics

Generative AI LLM Python SQL

Apply Now

Spotify

Media

Founded 2006

View All Jobs 26

Description

Define quality frameworks, evaluation rubrics, thresholds, and methodologies for multilingual AI experiences.
Design and execute structured evaluations for AI-generated, AI-translated, AI-curated, and recommendation-driven experiences.
Lead multilingual dataset curation, annotation, enrichment, and ground-truth creation to support AI model development and evaluation.
Analyze evaluation results, identify quality gaps, and provide actionable recommendations to improve multilingual AI quality.
Support LLM-as-a-judge workflows, evaluator calibration, and human-AI agreement studies.
Partner with Product, Engineering, Data Science, Research, Localization, vendors, and market experts to improve AI quality signals and inform launch decisions.
Document best practices and help define quality standards across languages, markets, and AI use cases.
Contribute to building scalable evaluation capabilities for future AI-powered experiences across Spotify.

Requirements

Experience in multilingual quality evaluation, localization, data curation, annotation, AI evaluation, or related fields, including text-to-text and text-to-speech experiences.
Understanding of language quality, cultural relevance, content quality, and user experience across multiple languages and markets.
Experience designing or conducting structured evaluations using quality rubrics, audits, annotation projects, or review methodologies.
Comfort using qualitative and quantitative data to identify trends, measure quality, and make recommendations.
Familiarity with large language models (LLMs), generative AI evaluation, human-in-the-loop workflows, or LLM-as-a-judge methodologies.
Ability to work through ambiguity and turn complex quality challenges into practical evaluation strategies.
Strong communication skills and experience collaborating in highly cross-functional environments with technical and non-technical partners.
Experience with recommendation systems, personalization, search, ranking, machine translation, generative AI, dataset creation, annotation operations, evaluator calibration, prompt testing, model evaluation, SQL, Python, dashboards, or annotation platforms is a plus.
Role is based in London or Stockholm.
Ability to work with some in-person meetings while maintaining flexibility to work from home.

Benefits

Flexible work environment with the ability to work from home.
Some in-person meetings with flexibility to work where you work best.
Reasonable accommodations available during the interview process and application stages.
Equal opportunity employer with an inclusive workplace culture.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Mid-Level Instructional Designer (Contract, Remote)

INFUSE 251-1K Professional Services

INFUSE is hiring an Instructional Designer to create blended, AI-enhanced learning programs for adult professionals within its global e-learning and demand generation organization.

Serbia Contract Mid Level AI (Artificial Intelligence) Instructional Designer

Generative AI LLM

13 hours, 55 minutes ago

Apply

13 hours, 55 minutes ago

Mid-Level Instructional Designer (Contract, Remote)

INFUSE 251-1K Professional Services

INFUSE is hiring an Instructional Designer to create blended, AI-enhanced learning experiences for adult professionals in support of its internal and external online education programs.

Ukraine Contract Mid Level AI (Artificial Intelligence) Instructional Designer

Generative AI LLM

13 hours, 55 minutes ago

Apply

13 hours, 55 minutes ago

Mid-Level Instructional Designer (Contract, Remote)

INFUSE 251-1K Professional Services

INFUSE is hiring an Instructional Designer to create blended, AI-enhanced e-learning programs for adult professionals in collaboration with subject matter experts and stakeholders.