Senior AI QA Automation Engineer

3 hours, 27 minutes ago
Senior
Artificial Intelligence and Machine Learning
Coforge

Coforge

Coforge is a global digital services provider specializing in transforming businesses through technology and industry expertise, powering growth with innovative solutions and platforms.

IT Services
10K-50K
Founded 1992

Description

  • Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs.
  • Create benchmark datasets, golden responses, regression suites, evaluation metrics, and release quality thresholds.
  • Test AI behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
  • Develop agent-powered solutions for test-case generation, API and workflow validation, and intelligent UAT.
  • Build UI, API, integration, performance, regression, and end-to-end test automation.
  • Implement CI/CD quality gates and monitor production AI quality, drift, reliability, and recurring failure patterns.
  • Translate business requirements, workflows, and AI behaviors into measurable quality criteria.

Requirements

  • 5+ years of experience in software quality engineering, test automation, or reliability engineering.
  • Strong Python skills and experience building reusable automated testing frameworks.
  • Hands-on experience with Playwright and/or Selenium.
  • Experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.
  • Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.
  • Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.
  • Knowledge of AI quality concepts including hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.
  • Experience implementing CI/CD quality gates and production monitoring.
  • Experience testing Generative AI, RAG solutions, copilots, or AI agents is preferred.
  • Experience with LLM evaluation frameworks, advanced prompt testing, adversarial testing, Responsible AI testing, or agent-powered QA solutions is preferred.
  • Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field.
  • Must be located in Costa Rica, Peru, Colombia, or Bolivia and able to work remotely.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Medical Director, AI Safety

Avo 11-50 Internet Software & Services

Avo is seeking a Medical Director, AI Safety to oversee the safety validation of AI-generated clinical content, establish release standards, and translate evaluation findings into product improvements while supporting clinical customer conversations.

3 hours, 42 minutes ago

Freelance Mechanical CAD&Simulation Engineer - AI Trainer

Mindrift.ai: Be the “I” in AI Internet Software & Services

Freelance mechanical or aerospace engineering opportunity to train and evaluate AI agents in real CAD environments by creating engineering geometries, reference solutions, and assessments for long-duration design tasks.

GitHub Python
3 hours, 57 minutes ago

Staff QA Automation Engineer

Meeru AI Internet Software & Services

Meeru AI is seeking a Staff Data Engineer to build and own production-grade pipelines, connectors, transformations, reconciliation, and lineage systems powering its auditable AI-native finance platform across multi-cloud customer environments.

Agile AWS Azure Burp Suite Cypress Datadog Docker EC2 FastAPI GCP GitHub Actions GitLab CI GraphQL HIPAA JavaScript Jenkins JIRA JMeter K6 Kubernetes Locust New Relic Next.js Playwright PostgreSQL Postman Prometheus Pytest Python React REST API Scrum SonarQube SQL SQLAlchemy TypeScript WebSockets
3 hours, 57 minutes ago

Lead/Architect GenAI Adoption (freelance, part-time)

N-iX 1K-5K Internet Software & Services

N-iX is seeking a GenAI Adoption Lead/AI Architect to help a global healthcare organization transform its software development lifecycle through enterprise AI architecture, agentic engineering, and measurable pilot implementations.

CI/CD Confluence Generative AI JIRA LLM
1 day, 2 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers