RemoteLeaf Logo
Log in Sign up

Data Quality Engineer, AI Business

3 months ago
North America
Full-time
Senior
QA Engineer
Data Science and Analytics
Python SQL Statistics
Apply Now
Prolific

Prolific

Prolific is a platform that enables researchers to quickly find trustworthy research participants. With a pool of over 120,000 active and verified participants, Prolific ensures high-quality responses through continuous monitoring and engagement. The p...

Professional Services
51-250
Founded 1997
$0M raised
View All Jobs 19

Description

  • Own end-to-end quality design for managed service studies, including rubrics, acceptance criteria, defect taxonomies, severity models, and definitions of done.
  • Define, implement, and maintain quality measurement systems such as sampling plans, golden sets, calibration protocols, agreement targets, adjudication workflows, and drift detection.
  • Build and deploy automated quality checks and launch gates using Python and SQL, including validation, completeness checks, anomaly detection, consistency testing, and label distribution monitoring.
  • Design and run launch readiness processes, including pre-launch checks, pilot calibration, ramp criteria, full-launch thresholds, and pause or rollback mechanisms.
  • Partner with Product and Engineering to embed quality controls and authenticity checks into workflows, tooling, and escalation paths.
  • Write and continuously improve guidelines and training materials for participants, reviewers, and internal teams.
  • Investigate quality and integrity issues end to end, perform root-cause analysis, and drive corrective and preventive actions.
  • Build dashboards and operating cadences to track defect rates, rework, throughput-quality trade-offs, integrity events, and SLA adherence.
  • Lead calibration sessions and coach QA leads and reviewers to improve decision consistency and rubric application.
  • Translate one-off fixes into repeatable, scalable playbooks across customers, programs, and study types.

Requirements

  • 5+ years of experience in quality engineering, data or annotation quality, analytics engineering, trust and integrity, or ML/LLM evaluation operations.
  • Strong proficiency in Python and SQL.
  • Comfort applying statistical concepts such as sampling strategies, confidence levels, and agreement metrics.
  • Proven ability to turn ambiguous or messy quality problems into clear metrics, automated checks, and durable process improvements.
  • Strong systems thinking with the ability to translate complex edge cases into clear rules, tests, rubrics, and governance mechanisms.
  • Hands-on experience instrumenting workflows and implementing pragmatic automation that catches quality and integrity issues early.
  • Demonstrated ability to influence cross-functional teams, including Product, Engineering, Operations, and Client teams, without direct authority.
  • Strong customer empathy and understanding of what trustworthy data means for research, AI training, and evaluation use cases.
  • Familiarity with data collection mechanics such as screeners, quota or routing constraints, and study design patterns is preferred.
  • Experience with LLM evals, red teaming, or policy-based annotation is preferred.
  • Data/versioning discipline, including dataset lineage, change control, and reproducibility, is preferred.
  • Experience with integrity or fraud detection systems and anti-abuse tooling is preferred.

Benefits

  • Competitive salary.
  • Equity as part of the compensation package.
  • Benefits package for eligible roles.
  • Remote working.
  • Potential cash variable compensation such as a bonus or commission.
  • Salary range provided for each role based on location, skills, experience, and relevant education or training.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Welo Global

Expert Rater Project - German (Spain)

Welo Global Professional Services

Welo Data is hiring an Expert Rater in Spain to coach raters, support quality evaluation workflows, and help improve the accuracy of AI-powered advertising systems.

Spain Freelance Junior QA Engineer Training Specialist
$41k-$41k
1 hour, 6 minutes ago
FSAStore.com

Sr. QA Engineer

FSAStore.com 51-250 Health Care Providers & Services

Health-E Commerce is hiring a Senior QA Engineer to oversee quality across multiple digital projects for its e-commerce healthcare brands, ensuring issues are caught before production.

United States Full-time Senior QA Engineer
$100k-$125k
Agile Android BDD GitHub Actions iOS Jenkins JIRA Playwright TypeScript
1 hour, 21 minutes ago
Welo Global

Expert Rater Project - Italian (Spain)

Welo Global Professional Services

Welo Data is hiring an Expert Rater in Spain to support Italian-language rating projects and help improve the quality of AI-powered advertising systems.

Spain Freelance Junior QA Engineer Training Specialist
$41k-$41k
1 hour, 21 minutes ago
On.Energy

Quality Assurance Engineer - Manual Testing

On.Energy 51-250 Electric Utilities

ON.energy is hiring a Mexico-based Quality Assurance Engineer to support ERP systems and business applications through testing, requirements analysis, UAT, and service desk collaboration.

Mexico Full-time Mid Level Manual QA Tester QA Engineer
ERP Microsoft Dynamics 365 Oracle SAP
1 hour, 36 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers