RemoteLeaf Logo
Log in Sign up

Senior AI Systems Quality Engineer

3 weeks, 1 day ago
United States
Full-time
Senior
AI (Artificial Intelligence)
Artificial Intelligence and Machine Learning
AWS CI/CD Databricks Datadog Grafana MLflow Prometheus Python TypeScript
Apply Now
Abacus Insights

Abacus Insights

Abacus Insights simplifies healthcare data with intelligent solutions, unlocking data value and empowering health plans, consumers, and providers.

Insurance
51-250
Founded 2017
$82M raised
View All Jobs 10

Description

  • Build production-grade validation frameworks, test harnesses, and evaluation pipelines across the AI lifecycle.
  • Design and evolve an AI testing platform integrated with Databricks and MLflow for repeatable testing, traceability, and auditability.
  • Create large-scale scenario-based test suites covering edge cases, long-tail scenarios, and failure modes.
  • Validate agent orchestration, including tool use, memory, decision logic, and non-deterministic behavior.
  • Define system contracts, guardrails, safe-degradation patterns, and measurable quality signals.
  • Integrate automated AI validation and quality gates into CI/CD pipelines for model, prompt, and code changes.
  • Develop reusable libraries and components for consistent AI quality practices.
  • Own aspects of AI release readiness and define go/no-go criteria using measurable thresholds.
  • Collaborate with AI, platform, security, domain, and delivery teams to establish production-relevant quality criteria.

Requirements

  • 7+ years of software engineering experience, primarily in backend or platform systems.
  • Production experience designing and implementing AI testing automation and custom validation frameworks.
  • Strong proficiency in Python and/or TypeScript.
  • Hands-on experience with LLM-based or agentic systems and non-deterministic behavior.
  • Experience with large-scale regression testing, long-tail evaluation, and broad test coverage.
  • Deep understanding of CI/CD integration and automated deployment quality gates.
  • Solid understanding of AWS cloud-native architectures.
  • Experience engineering for reliability, governance, safety, security, privacy, and operational risk in regulated or mission-critical environments.
  • Experience with AI evaluation methods such as drift detection, bias and fairness testing, hallucination assessment, and regression strategies.
  • Ability to define enforceable AI trust thresholds, including accuracy, hallucination limits, explainability, and PHI-safe behavior.
  • Experience working with domain experts to define correctness and production-relevant validation scenarios.
  • Preferred: Databricks Medallion architecture, MLflow, observability tools, LLM guardrails, performance and cost regression testing, AI/ML certifications, prompt and agent orchestration testing, or AI-generated test scenarios.

Benefits

  • Base salary plus eligibility for performance bonuses and equity grants.
  • Unlimited paid time off.
  • Work-from-anywhere flexibility.
  • Comprehensive health coverage with multiple plan options.
  • Equity for every employee.
  • Growth-focused professional development environment.
  • Home office setup allowance.
  • Monthly cell phone allowance.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Metova

Automation Test Engineer

Metova 51-250 Internet Software & Services

Senior Automation Test Engineer responsible for building quality engineering practices and automated testing strategies for web applications, REST APIs, and AI-powered solutions.

Argentina Bolivia Chile Colombia Peru Venezuela Contract Senior SDET (Software Development Engineer in Test)
Azure CI/CD Gatling Git Java JavaScript Playwright Python REST API Selenium TypeScript
6 hours, 28 minutes ago
Mindrift.ai: Be the “I” in AI

Freelance Mechanical CAD&Simulation Engineer - AI Trainer

Mindrift.ai: Be the “I” in AI Internet Software & Services

Freelance Mechanical or Aerospace Engineering contributor helping train and evaluate AI agents in a real CAD environment by creating engineering geometry, reference solutions, and assessments for complex tasks.

Bosnia and Herzegovina Estonia Latvia Lithuania Malta Serbia Freelance Senior AI (Artificial Intelligence)
Up to $64k
GitHub Python
6 hours, 28 minutes ago
Freedom24

Fullstack QA Engineer

Freedom24 Capital Markets

Join a banking QA team as a Fullstack QA Engineer, testing mobile banking applications and shared backend services that support cards, transfers, and payments.

Serbia Full-time Mid Level QA Engineer SDET (Software Development Engineer in Test)
Android Appium Espresso Firebase Git GitLab CI HTTP iOS Java Kotlin Newman Postman REST API SQL
6 hours, 43 minutes ago
Weekday

AI Safety Experts - English & Finnish

Weekday 11-50 Construction & Engineering

A client is seeking a bilingual AI Red Team Specialist to probe conversational AI systems, identify vulnerabilities, and produce actionable data that improves model safety.

United States Contract Mid Level AI (Artificial Intelligence)
$100k-$129k
Cybersecurity Penetration Testing
6 hours, 43 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers