RemoteLeaf Logo
Log in Sign up

Data Quality Engineer, AI Business

6 months ago
North America
Full-time
Senior
QA Engineer
Data Science and Analytics
Python SQL Statistics
Apply Now
Prolific

Prolific

Prolific is a platform that enables researchers to quickly find trustworthy research participants. With a pool of over 120,000 active and verified participants, Prolific ensures high-quality responses through continuous monitoring and engagement. The p...

Professional Services
51-250
Founded 1997
$0M raised
View All Jobs 15

Description

  • Own end-to-end quality design for managed service studies, including rubrics, acceptance criteria, defect taxonomies, severity models, and definitions of done.
  • Define, implement, and maintain quality measurement systems such as sampling plans, golden sets, calibration protocols, agreement targets, adjudication workflows, and drift detection.
  • Build and deploy automated quality checks and launch gates using Python and SQL, including validation, completeness checks, anomaly detection, consistency testing, and label distribution monitoring.
  • Design and run launch readiness processes, including pre-launch checks, pilot calibration, ramp criteria, full-launch thresholds, and pause or rollback mechanisms.
  • Partner with Product and Engineering to embed quality controls and authenticity checks into workflows, tooling, and escalation paths.
  • Write and continuously improve guidelines and training materials for participants, reviewers, and internal teams.
  • Investigate quality and integrity issues end to end, perform root-cause analysis, and drive corrective and preventive actions.
  • Build dashboards and operating cadences to track defect rates, rework, throughput-quality trade-offs, integrity events, and SLA adherence.
  • Lead calibration sessions and coach QA leads and reviewers to improve decision consistency and rubric application.
  • Translate one-off fixes into repeatable, scalable playbooks across customers, programs, and study types.

Requirements

  • 5+ years of experience in quality engineering, data or annotation quality, analytics engineering, trust and integrity, or ML/LLM evaluation operations.
  • Strong proficiency in Python and SQL.
  • Comfort applying statistical concepts such as sampling strategies, confidence levels, and agreement metrics.
  • Proven ability to turn ambiguous or messy quality problems into clear metrics, automated checks, and durable process improvements.
  • Strong systems thinking with the ability to translate complex edge cases into clear rules, tests, rubrics, and governance mechanisms.
  • Hands-on experience instrumenting workflows and implementing pragmatic automation that catches quality and integrity issues early.
  • Demonstrated ability to influence cross-functional teams, including Product, Engineering, Operations, and Client teams, without direct authority.
  • Strong customer empathy and understanding of what trustworthy data means for research, AI training, and evaluation use cases.
  • Familiarity with data collection mechanics such as screeners, quota or routing constraints, and study design patterns is preferred.
  • Experience with LLM evals, red teaming, or policy-based annotation is preferred.
  • Data/versioning discipline, including dataset lineage, change control, and reproducibility, is preferred.
  • Experience with integrity or fraud detection systems and anti-abuse tooling is preferred.

Benefits

  • Competitive salary.
  • Equity as part of the compensation package.
  • Benefits package for eligible roles.
  • Remote working.
  • Potential cash variable compensation such as a bonus or commission.
  • Salary range provided for each role based on location, skills, experience, and relevant education or training.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Assail

QA Engineer

Assail 1-10 information technology & services

QA Engineer responsible for validating web and API-focused software releases through structured and exploratory testing so customers receive reliable products.

United States Mid Level QA Engineer
JIRA Linear SQL
1 day, 2 hours ago
Hireframe

Market Research Analyst

Hireframe 51-250 Professional Services

Hireframe is seeking a Quantitative Research Operations Manager to support end-to-end quantitative research execution by ensuring surveys, data, analyses, and client reporting outputs are accurate and high quality.

Philippines Full-time Mid Level Data Analyst QA Engineer
1 day, 3 hours ago
Murj

Software Quality Assurance Engineer

Murj 11-50 Internet Software & Services

Murj is seeking an SQA Engineer to ensure the quality, reliability, and usability of its enterprise SaaS platform for managing implantable cardiac devices and supporting clinical care.

United States Full-time Mid Level QA Engineer
Agile HIPAA JIRA Postman Scrum
2 days, 2 hours ago
Clariti

QA Engineer, Agentic Development

Clariti 51-250 Internet Software & Services

Clariti is seeking a QA Engineer to define quality practices for a new product using agentic development, leading testing strategy, automation, and quality decisions across delivery.

Canada Full-time Senior QA Engineer
$75k-$90k
Agile CI/CD JavaScript JIRA Playwright Postman TypeScript
3 days, 3 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers