RemoteLeaf Logo
Log in Sign up

Senior AI Agent Evaluation Engineer

2 months, 2 weeks ago
Anywhere
Part-time
Senior
AI (Artificial Intelligence)
Artificial Intelligence and Machine Learning
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 21

Description

  • Build realistic virtual developer environments with codebases, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design tasks from intermediate states of the simulated environments and define what counts as a solved solution.
  • Create prompts that are challenging but solvable by an AI agent.
  • Write tests that validate agent solutions, accepting all valid approaches and rejecting incorrect ones.
  • Review agent solutions, analyze failures, and iterate on tasks and tests based on QA feedback.
  • Refine evaluation criteria until the task design is fair and robust.
  • Identify scenarios that reveal the strengths and weaknesses of frontier coding models.
  • Submit completed tasks by the deadline and meet the required acceptance criteria.

Requirements

  • 5+ years of software development experience.
  • Experience with Python and FastAPI.
  • Experience with JavaScript or TypeScript and React.
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2+ level.
  • Ability to work on a project-based, non-permanent basis.
  • Willingness to complete qualification steps before joining a project.

Benefits

  • Up to $50/hr equivalent compensation, depending on level and pace.
  • Flexible schedule with the ability to choose when and how to work.
  • Project-based work with estimated tasks of around 20 hours each.
  • Payment for completed tasks upon acceptance.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

GR8 Tech

Middle QA Engineer for Client Centric Team

GR8 Tech 251-1K IT Services

GR8_TECH is seeking a Manual QA Engineer to ensure reliable quality across its B2B iGaming Web, native Mobile, Backend, and integration products from feature analysis through release support.

Poland Full-time Mid Level QA Engineer
Android CI/CD Docker Git HTTP iOS Kafka Kubernetes Microservices OpenAPI Postman SQL TLS
8 hours, 56 minutes ago
Caseware

Quality Assurance Engineer

Caseware 251-1K Internet Software & Services

Caseware is hiring a remote Senior Software Development Engineer in Test to improve automated and manual testing for its Caseware IDEA Windows desktop application used by audit, accounting, and finance professionals worldwide.

Romania Full-time Senior QA Engineer
C# CI/CD GitHub Playwright Postman PowerShell Python SQL
9 hours, 26 minutes ago
Pavago

Quality Assurance Engineer

Pavago IT Services

Pavago is hiring a remote Quality Assurance Engineer for a client to combine manual and automated testing with defect management, performance validation, and release quality assurance for SaaS applications.

Pakistan Full-time Junior QA Engineer
Agile Azure CI/CD Cypress GitHub Actions GitLab CI Jenkins JIRA Locust Playwright Postman Selenium
9 hours, 41 minutes ago
Appfire

Manager, QA Engineering

Appfire 251-1K Internet Software & Services

Appfire is hiring a remote Manager of Quality Assurance to lead QA engineers, improve automation and quality processes, and ensure reliable software delivery across its product portfolio.

Poland Full-time Lead QA Engineer
AWS CI/CD Cypress Docker GitHub GitLab Jenkins JMeter Kubernetes Selenium
1 day, 9 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers