RemoteLeaf Logo
Log in Sign up

Data Quality Engineer, AI Business

9 hours, 51 minutes ago
North America
Full-time
Senior
QA Engineer
Data Science and Analytics
Python SQL Statistics
Apply Now
Prolific

Prolific

Prolific is a platform that enables researchers to quickly find trustworthy research participants. With a pool of over 120,000 active and verified participants, Prolific ensures high-quality responses through continuous monitoring and engagement. The p...

Professional Services
51-250
Founded 1997
$0M raised
View All Jobs 158

Description

  • Own end-to-end quality design for managed service studies, including rubrics, acceptance criteria, defect taxonomies, severity models, and definitions of done.
  • Define, implement, and maintain quality measurement systems such as sampling plans, golden sets, calibration protocols, agreement targets, adjudication workflows, and drift detection.
  • Build and deploy automated quality checks and launch gates using Python and SQL, including validation, completeness checks, anomaly detection, consistency testing, and label distribution monitoring.
  • Design and run launch readiness processes, including pre-launch checks, pilot calibration, ramp criteria, full-launch thresholds, and pause or rollback mechanisms.
  • Partner with Product and Engineering to embed quality controls and authenticity checks into workflows, tooling, and escalation paths.
  • Write and continuously improve guidelines and training materials for participants, reviewers, and internal teams.
  • Investigate quality and integrity issues end to end, perform root-cause analysis, and drive corrective and preventive actions.
  • Build dashboards and operating cadences to track defect rates, rework, throughput-quality trade-offs, integrity events, and SLA adherence.
  • Lead calibration sessions and coach QA leads and reviewers to improve decision consistency and rubric application.
  • Translate one-off fixes into repeatable, scalable playbooks across customers, programs, and study types.

Requirements

  • 5+ years of experience in quality engineering, data or annotation quality, analytics engineering, trust and integrity, or ML/LLM evaluation operations.
  • Strong proficiency in Python and SQL.
  • Comfort applying statistical concepts such as sampling strategies, confidence levels, and agreement metrics.
  • Proven ability to turn ambiguous or messy quality problems into clear metrics, automated checks, and durable process improvements.
  • Strong systems thinking with the ability to translate complex edge cases into clear rules, tests, rubrics, and governance mechanisms.
  • Hands-on experience instrumenting workflows and implementing pragmatic automation that catches quality and integrity issues early.
  • Demonstrated ability to influence cross-functional teams, including Product, Engineering, Operations, and Client teams, without direct authority.
  • Strong customer empathy and understanding of what trustworthy data means for research, AI training, and evaluation use cases.
  • Familiarity with data collection mechanics such as screeners, quota or routing constraints, and study design patterns is preferred.
  • Experience with LLM evals, red teaming, or policy-based annotation is preferred.
  • Data/versioning discipline, including dataset lineage, change control, and reproducibility, is preferred.
  • Experience with integrity or fraud detection systems and anti-abuse tooling is preferred.

Benefits

  • Competitive salary.
  • Equity as part of the compensation package.
  • Benefits package for eligible roles.
  • Remote working.
  • Potential cash variable compensation such as a bonus or commission.
  • Salary range provided for each role based on location, skills, experience, and relevant education or training.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

SupplyHouse.com

QA Engineer

SupplyHouse.com 251-1K Building Materials

SupplyHouse.com is hiring a remote QA Engineer in India to support its e-commerce IT team by ensuring software quality across web and mobile applications through automated and collaborative testing.

India Full-time Mid Level QA Engineer
$18k-$23k
Appium Chai Cypress Java JavaScript Microservices Mocha MySQL Postman REST API Selenium SQL
5 hours, 21 minutes ago
ClickHouse

QA Engineer - Core Database (remote)

ClickHouse 51-250 IT Services

ClickHouse is hiring a Senior Software Quality Engineer to improve the quality and release reliability of its open-source database and cloud service at global scale.

United States Full-time Senior QA Engineer SDET (Software Development Engineer in Test)
$141k-$230k
C++ Git Go Linux Python Shell Scripting SQL
8 hours, 36 minutes ago
SpaceX

CMM Source Inspector (Travel) - Chicago

SpaceX 10K-50K Aerospace & Defense

SpaceX is hiring a CMM Source Inspector in Chicago to travel to supplier sites and verify the quality and conformance of manufactured aerospace parts, assemblies, and flight hardware.

United States Full-time Mid Level QA Engineer
$62k-$100k
9 hours, 36 minutes ago
SpaceX

CMM Source Inspector (Remote) - Chicago

SpaceX 10K-50K Aerospace & Defense

SpaceX is hiring a Remote CMM Source Inspector in Chicago to travel to supplier sites and inspect manufactured flight hardware and mechanical components against engineering and quality requirements.

United States Full-time Mid Level QA Engineer
$62k-$100k
9 hours, 51 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers