RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

10 hours, 8 minutes ago
Canada
Contract
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 183

Description

  • Build realistic developer environments, including a virtual company codebase, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design tasks from intermediate states of these environments and define what counts as a solved solution.
  • Create clear evaluation criteria and prompts that make tasks solvable by an AI agent.
  • Write tests that verify agent solutions, accepting valid approaches and rejecting incorrect ones.
  • Review agent outputs, analyze failure cases, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation materials until the task set is fair, robust, and reliable.
  • Help assess whether AI coding agents can handle real-world developer tasks in believable scenarios.

Requirements

  • 5+ years of experience in software development.
  • Experience with the core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
  • Experience writing tests, including functional and integration tests.
  • English proficiency at B2 level or higher.
  • Submit a CV in English.
  • Ability to complete project tasks within the estimated 20-hour effort window, by the stated deadline.
  • Understanding of how to distinguish strong and weak coding solutions in complex scenarios.
  • Participation is project-based and not permanent employment.

Benefits

  • Compensation of up to $50/hr equivalent, depending on level and pace.
  • Flexible schedule with self-directed working hours.
  • Project-based work rather than a permanent employment commitment.
  • Estimated 20-hour tasks, allowing candidates to choose when and how to work.
  • Payment after completing project tasks and meeting acceptance criteria.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Toloka

Record Your Daily Routine & Get Paid - AI Training (Remote)

Toloka 251-1K Internet Software & Services

Project-based freelance opportunity with an AI training platform recording first-person videos of everyday household activities to help train AI systems and robots.

Guatemala Freelance Entry Level AI (Artificial Intelligence)
$12k-$12k
9 hours, 53 minutes ago
Toloka

AI Trainer - Freelance Annotator (English)

Toloka 251-1K Internet Software & Services

Mindrift is seeking a project-based AI Trainer - Annotator to evaluate and improve AI responses for e-commerce and online shopping tasks on a non-permanent, part-time basis.

United States Part-time Entry Level AI (Artificial Intelligence) Manual QA Tester
Up to $37k
E-commerce Machine Learning
9 hours, 53 minutes ago
Weekday

Biology & Biophysics Researchers (India, Part-time)

Weekday 11-50 Construction & Engineering

An AI lab client is hiring part-time life science researchers to help train and evaluate frontier AI systems on advanced biological and biophysical reasoning.

India Part-time Lead AI (Artificial Intelligence) Research Scientist
$0k-$0k
Machine Learning
9 hours, 53 minutes ago
Toloka

Creador(a) de Contenido POV para IA - Remoto

Toloka 251-1K Internet Software & Services

Proyecto remoto y flexible con una empresa de entrenamiento de IA para grabar actividades cotidianas desde casa y ayudar a entrenar sistemas de IA y robótica.

Honduras Part-time Entry Level AI (Artificial Intelligence)
$12k-$12k
10 hours, 8 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers