RemoteLeaf Logo
Log in Sign up

Multilingual AI Quality Specialist

1 week, 3 days ago
Sweden, United Kingdom
Full-time
Mid Level
AI (Artificial Intelligence)
Data Science and Analytics
Generative AI LLM Python SQL
Apply Now
Spotify

Spotify

Spotify is a leading global audio streaming service empowering artists and inspiring fans worldwide with over 70 million tracks. It has 365 million users, including 165 million subscribers, across 178 markets.

Media
Founded 2006
View All Jobs 27

Description

  • Define quality frameworks, evaluation rubrics, thresholds, and methodologies for multilingual AI experiences.
  • Design and execute structured evaluations for AI-generated, AI-translated, AI-curated, and recommendation-driven experiences.
  • Lead multilingual dataset curation, annotation, enrichment, and ground-truth creation to support AI model development and evaluation.
  • Analyze evaluation results, identify quality gaps, and provide actionable recommendations to improve multilingual AI quality.
  • Support LLM-as-a-judge workflows, evaluator calibration, and human-AI agreement studies.
  • Partner with Product, Engineering, Data Science, Research, Localization, vendors, and market experts to improve AI quality signals and inform launch decisions.
  • Document best practices and help define quality standards across languages, markets, and AI use cases.
  • Contribute to building scalable evaluation capabilities for future AI-powered experiences across Spotify.

Requirements

  • Experience in multilingual quality evaluation, localization, data curation, annotation, AI evaluation, or related fields, including text-to-text and text-to-speech experiences.
  • Understanding of language quality, cultural relevance, content quality, and user experience across multiple languages and markets.
  • Experience designing or conducting structured evaluations using quality rubrics, audits, annotation projects, or review methodologies.
  • Comfort using qualitative and quantitative data to identify trends, measure quality, and make recommendations.
  • Familiarity with large language models (LLMs), generative AI evaluation, human-in-the-loop workflows, or LLM-as-a-judge methodologies.
  • Ability to work through ambiguity and turn complex quality challenges into practical evaluation strategies.
  • Strong communication skills and experience collaborating in highly cross-functional environments with technical and non-technical partners.
  • Experience with recommendation systems, personalization, search, ranking, machine translation, generative AI, dataset creation, annotation operations, evaluator calibration, prompt testing, model evaluation, SQL, Python, dashboards, or annotation platforms is a plus.
  • Role is based in London or Stockholm.
  • Ability to work with some in-person meetings while maintaining flexibility to work from home.

Benefits

  • Flexible work environment with the ability to work from home.
  • Some in-person meetings with flexibility to work where you work best.
  • Reasonable accommodations available during the interview process and application stages.
  • Equal opportunity employer with an inclusive workplace culture.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

AvePoint

Sr. Product Marketing Manager- AI & Data Security

AvePoint 1K-5K Internet Software & Services

AvePoint is seeking a Senior Product Marketing Manager to help shape go-to-market strategy for its AI security solutions and drive growth at a high-growth public company.

United States Full-time Senior AI (Artificial Intelligence) Product Marketing Manager
$150k-$190k
13 hours, 34 minutes ago
INFUSE

Production AI Innovation Lead (Contract, Remote)

INFUSE 251-1K Professional Services

INFUSE Academy is seeking a design-and-AI-focused learning experience creator to develop and scale new production workflows for course and asset creation.

Ukraine Contract Senior AI (Artificial Intelligence) Design Lead
Figma Illustrator Photoshop
13 hours, 49 minutes ago
INFUSE

Production AI Innovation Lead (Contract, Remote)

INFUSE 251-1K Professional Services

INFUSE Academy is hiring a learning experience designer to create and scale AI-informed workflows for producing courses and learning assets.

Portugal Contract Senior AI (Artificial Intelligence) Design Lead
Figma Illustrator Photoshop
14 hours, 4 minutes ago
On.Energy

Director of Program Management - AI Enablement

On.Energy 51-250 Electric Utilities

ON.energy is hiring a Director of Program Management, AI Enablement to build and run its enterprise AI adoption program, helping the company turn AI use into measurable business impact across the organization.

United States Full-time Lead AI (Artificial Intelligence) Program Manager
ERP
14 hours, 4 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers