RemoteLeaf Logo
Log in Sign up

Data Quality Engineer, AI Business

3 months, 3 weeks ago
North America
Full-time
Senior
QA Engineer
Data Science and Analytics
Python SQL Statistics
Apply Now
Prolific

Prolific

Prolific is a platform that enables researchers to quickly find trustworthy research participants. With a pool of over 120,000 active and verified participants, Prolific ensures high-quality responses through continuous monitoring and engagement. The p...

Professional Services
51-250
Founded 1997
$0M raised
View All Jobs 12

Description

  • Own end-to-end quality design for managed service studies, including rubrics, acceptance criteria, defect taxonomies, severity models, and definitions of done.
  • Define, implement, and maintain quality measurement systems such as sampling plans, golden sets, calibration protocols, agreement targets, adjudication workflows, and drift detection.
  • Build and deploy automated quality checks and launch gates using Python and SQL, including validation, completeness checks, anomaly detection, consistency testing, and label distribution monitoring.
  • Design and run launch readiness processes, including pre-launch checks, pilot calibration, ramp criteria, full-launch thresholds, and pause or rollback mechanisms.
  • Partner with Product and Engineering to embed quality controls and authenticity checks into workflows, tooling, and escalation paths.
  • Write and continuously improve guidelines and training materials for participants, reviewers, and internal teams.
  • Investigate quality and integrity issues end to end, perform root-cause analysis, and drive corrective and preventive actions.
  • Build dashboards and operating cadences to track defect rates, rework, throughput-quality trade-offs, integrity events, and SLA adherence.
  • Lead calibration sessions and coach QA leads and reviewers to improve decision consistency and rubric application.
  • Translate one-off fixes into repeatable, scalable playbooks across customers, programs, and study types.

Requirements

  • 5+ years of experience in quality engineering, data or annotation quality, analytics engineering, trust and integrity, or ML/LLM evaluation operations.
  • Strong proficiency in Python and SQL.
  • Comfort applying statistical concepts such as sampling strategies, confidence levels, and agreement metrics.
  • Proven ability to turn ambiguous or messy quality problems into clear metrics, automated checks, and durable process improvements.
  • Strong systems thinking with the ability to translate complex edge cases into clear rules, tests, rubrics, and governance mechanisms.
  • Hands-on experience instrumenting workflows and implementing pragmatic automation that catches quality and integrity issues early.
  • Demonstrated ability to influence cross-functional teams, including Product, Engineering, Operations, and Client teams, without direct authority.
  • Strong customer empathy and understanding of what trustworthy data means for research, AI training, and evaluation use cases.
  • Familiarity with data collection mechanics such as screeners, quota or routing constraints, and study design patterns is preferred.
  • Experience with LLM evals, red teaming, or policy-based annotation is preferred.
  • Data/versioning discipline, including dataset lineage, change control, and reproducibility, is preferred.
  • Experience with integrity or fraud detection systems and anti-abuse tooling is preferred.

Benefits

  • Competitive salary.
  • Equity as part of the compensation package.
  • Benefits package for eligible roles.
  • Remote working.
  • Potential cash variable compensation such as a bonus or commission.
  • Salary range provided for each role based on location, skills, experience, and relevant education or training.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Robusta Technology Group

Senior QA Engineer - Salesforce | 3-Month Contract | Remote | Immediate Hiring

Robusta Technology Group 51-250 Professional Services

Octopus, part of Robusta Technology Group, is hiring a Senior QA Engineer for a 3-month remote contract to ensure the quality and reliability of Salesforce-based solutions across the development lifecycle.

Egypt Contract Senior QA Engineer Salesforce Developer
Agile CI/CD Salesforce
13 hours, 35 minutes ago
Mindrift.ai: Be the “I” in AI

AI Evaluation Engineer (Python, QA or Security)

Mindrift.ai: Be the “I” in AI Internet Software & Services

Mindrift is seeking experienced software developers for project-based work creating realistic coding evaluation tasks and tests for AI agents.

United States Part-time Senior AI (Artificial Intelligence) QA Engineer
$104k-$104k
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
13 hours, 35 minutes ago
Kyivstar

QA Engineer (project TutTam)

Kyivstar 1K-5K Wireless Telecommunication Services

Kyivstar.Tech is hiring a QA Engineer for the TutTam FoodTech marketplace to ensure quality across its mobile app and web product throughout the development lifecycle.

Ukraine Full-time Junior QA Engineer
Agile Android CI/CD Confluence Firebase iOS JIRA Kanban Postman REST API Scrum SQL
1 day, 13 hours ago
Planner 5D

QA Engineer (Manual)

Planner 5D 51-250 Internet Software & Services

Planner 5D is hiring a Manual Quality Assurance Engineer to help improve the quality of its cross-platform home design product across web, mobile, and desktop applications.

Latvia Armenia Serbia Georgia Spain Estonia Contract Mid Level QA Engineer
Android Android Studio GitLab iOS macOS Postman Xcode
1 day, 13 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers