RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

2 hours, 29 minutes ago
Canada
Freelance
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 353

Description

  • Build realistic developer environments that simulate a virtual company, including codebase, infrastructure, tickets, documentation, and conversation history.
  • Design coding tasks from intermediate states of those environments and define clear criteria for what counts as a solved task.
  • Write tests that validate AI agent solutions, accepting all correct approaches while rejecting incorrect ones.
  • Review agent outputs, analyze failures, and iterate on tasks and tests based on QA feedback.
  • Ensure tasks are challenging for frontier AI coding models and grounded in realistic development scenarios.
  • Collaborate within a project-based workflow to complete tasks and submit them by the required deadline.

Requirements

  • 5+ years of experience in software development.
  • Experience with Python, including FastAPI.
  • Experience with JavaScript or TypeScript, including React.
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Ability to work on a project-based, non-permanent contract.
  • Willingness to submit a CV in English and indicate English proficiency level.

Benefits

  • Up to $50/hr equivalent compensation, depending on level and pace.
  • Project-based flexibility with tasks completed on your own schedule.
  • Estimated 20 hours per task, depending on complexity.
  • Opportunity to work on AI evaluation projects for leading tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Esri

Software Development Engineer

Esri 1K-5K Internet Software & Services

Esri is hiring a software testing professional to help build, automate, and maintain integration and regression test frameworks for AI, GIS, and web-based software solutions.

United States Full-time Junior Software Engineer
$137k-$140k
CI/CD Java Jenkins JUnit Python Selenium
1 hour, 29 minutes ago
Cordial

Senior Software Engineer

Cordial 51-250 Media

Cordial is hiring a Senior Software Engineer to lead development of scalable, high-performance systems for its marketing platform and guide engineering direction across the product.

United States Full-time Senior Software Engineer
$160k-$175k
System Design
1 hour, 29 minutes ago
Esri

Software Development Engineer

Esri 1K-5K Internet Software & Services

Esri is hiring a software engineer to help build and enhance ArcGIS Online and Enterprise, delivering scalable geospatial and analytics products across cloud and on-premises environments.

United States Full-time Entry Level Software Engineer
$137k-$140k
Java Microservices Python
1 hour, 44 minutes ago
Prolific

AI Trainer - Kannada - Seattle, US

Prolific 51-250 Professional Services

Prolific is hiring remote, task-based AI Trainers fluent in Kannada to evaluate text and audio snippets and help improve how AI understands natural language.

Anywhere Contract Entry Level AI (Artificial Intelligence)
$7k-$7k
1 hour, 59 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers