RemoteLeaf Logo
Log in Sign up

Data Quality Engineer, AI Business

4 months, 3 weeks ago
North America
Full-time
Senior
QA Engineer
Data Science and Analytics
Python SQL Statistics
Apply Now
Prolific

Prolific

Prolific is a platform that enables researchers to quickly find trustworthy research participants. With a pool of over 120,000 active and verified participants, Prolific ensures high-quality responses through continuous monitoring and engagement. The p...

Professional Services
51-250
Founded 1997
$0M raised
View All Jobs 12

Description

  • Own end-to-end quality design for managed service studies, including rubrics, acceptance criteria, defect taxonomies, severity models, and definitions of done.
  • Define, implement, and maintain quality measurement systems such as sampling plans, golden sets, calibration protocols, agreement targets, adjudication workflows, and drift detection.
  • Build and deploy automated quality checks and launch gates using Python and SQL, including validation, completeness checks, anomaly detection, consistency testing, and label distribution monitoring.
  • Design and run launch readiness processes, including pre-launch checks, pilot calibration, ramp criteria, full-launch thresholds, and pause or rollback mechanisms.
  • Partner with Product and Engineering to embed quality controls and authenticity checks into workflows, tooling, and escalation paths.
  • Write and continuously improve guidelines and training materials for participants, reviewers, and internal teams.
  • Investigate quality and integrity issues end to end, perform root-cause analysis, and drive corrective and preventive actions.
  • Build dashboards and operating cadences to track defect rates, rework, throughput-quality trade-offs, integrity events, and SLA adherence.
  • Lead calibration sessions and coach QA leads and reviewers to improve decision consistency and rubric application.
  • Translate one-off fixes into repeatable, scalable playbooks across customers, programs, and study types.

Requirements

  • 5+ years of experience in quality engineering, data or annotation quality, analytics engineering, trust and integrity, or ML/LLM evaluation operations.
  • Strong proficiency in Python and SQL.
  • Comfort applying statistical concepts such as sampling strategies, confidence levels, and agreement metrics.
  • Proven ability to turn ambiguous or messy quality problems into clear metrics, automated checks, and durable process improvements.
  • Strong systems thinking with the ability to translate complex edge cases into clear rules, tests, rubrics, and governance mechanisms.
  • Hands-on experience instrumenting workflows and implementing pragmatic automation that catches quality and integrity issues early.
  • Demonstrated ability to influence cross-functional teams, including Product, Engineering, Operations, and Client teams, without direct authority.
  • Strong customer empathy and understanding of what trustworthy data means for research, AI training, and evaluation use cases.
  • Familiarity with data collection mechanics such as screeners, quota or routing constraints, and study design patterns is preferred.
  • Experience with LLM evals, red teaming, or policy-based annotation is preferred.
  • Data/versioning discipline, including dataset lineage, change control, and reproducibility, is preferred.
  • Experience with integrity or fraud detection systems and anti-abuse tooling is preferred.

Benefits

  • Competitive salary.
  • Equity as part of the compensation package.
  • Benefits package for eligible roles.
  • Remote working.
  • Potential cash variable compensation such as a bonus or commission.
  • Salary range provided for each role based on location, skills, experience, and relevant education or training.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Pingwind

QA Engineer

Pingwind 51-250 Internet Software & Services

PingWind is seeking a Lead QA Engineer to oversee software quality assurance for federal government applications, designing automated testing and validating AWS-based systems through Azure DevOps.

United States Full-time Lead QA Engineer
$90k-$124k
Agile Appium AWS Azure CI/CD Cybersecurity Cypress JIRA Microservices Selenium
1 day, 3 hours ago
CI&T

[Job-31544] Senior QA, Brazil

CI&T 5K-10K Internet Software & Services

CI&T is seeking an experienced QA Tester to ensure the quality of a new CMS-powered website platform and its APIs for an enterprise client through manual, automated, and AI-supported testing.

Brazil Full-time Senior QA Engineer
Agile CI/CD Git GitFlow Postman REST API Sanity Selenium
1 day, 4 hours ago
Filevine

Senior QA Analyst - Auth & Auth

Filevine 251-1K Specialized Consumer Services

Filevine is seeking a senior QA professional to lead quality initiatives for AI-driven legal workflows and integrations while partnering with Product and Engineering to deliver reliable, compliant, scalable releases.

United States Full-time Lead QA Engineer
$140k-$170k
1 day, 4 hours ago
Dev.Pro

Intermediate Test Engineer (Manual/Automation) - OP02239

Dev.Pro 251-1K Internet Software & Services

Dev.Pro is hiring an Intermediate Test Engineer in Latin America to test and automate functionality across the UI, backend, and integration layers of a modern payments platform for a global fintech client.

Colombia Argentina Brazil Full-time Mid Level QA Engineer
C# Confluence Datadog JIRA .NET Playwright Postman
1 day, 4 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers