RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

12 hours, 38 minutes ago
Philippines
Part-time
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 376

Description

  • Build realistic virtual developer environments with codebase, infrastructure, tickets, documentation, and conversation history.
  • Design tasks from intermediate environment states, including the prompt and clear definitions of what counts as a solved solution.
  • Write functional and integration tests that validate correct AI agent solutions and reject incorrect ones.
  • Review agent outputs, analyze failures, and refine tasks and tests based on QA feedback.
  • Iterate until the evaluation setup is fair, robust, and able to challenge strong AI coding models.
  • Ensure tasks are solvable by an AI agent and allow multiple valid implementation approaches.
  • Assess whether solutions meet the listed acceptance criteria before submission.

Requirements

  • 5+ years of software development experience.
  • Experience with the core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2+ level.
  • Ability to work on a project-based, non-permanent engagement.
  • Ability to complete tasks within an estimated 20 hours per project and submit by the deadline.
  • Comfort working in a role where the agent writes most of the code and you guide and evaluate rather than build from scratch.

Benefits

  • Up to $30/hr equivalent compensation, depending on level and pace.
  • Flexible schedule with no fixed working hours; you choose when and how to work.
  • Project-based paid work with tasks completed to deadline.
  • Estimated ~20 hours of work per task/project.
  • Opportunity to work on leading-edge AI evaluation projects for tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Leading Educators

Director of AI-Enabled Instruction, ELA

Leading Educators 51-250 Diversified Consumer Services

Leading Educators is hiring a Director of AI-Enabled Instruction, ELA to lead a multi-school literacy initiative and advance its broader AI-enabled ELA strategy through curriculum-aligned instructional design, coaching, and systems support.

United States Full-time Lead AI (Artificial Intelligence) Instructional Designer
$104k-$140k
Prototyping
11 hours, 53 minutes ago
Remote Recruitment

AI Integration Specialist - Italian Speaking

Remote Recruitment 11-50 Professional Services

AI Integration Specialist role at an Italian dental business coaching and consulting organisation, focused on implementing AI and automation solutions that improve efficiency, data use, and business performance across the company.

South Africa Italy Full-time Mid Level AI (Artificial Intelligence) AI Engineer
LLM Machine Learning Python SQL Tableau
12 hours, 23 minutes ago
Toloka

AI Trainer - Freelance Data Annotator

Toloka 251-1K Internet Software & Services

Toloka is hiring freelance annotators to support Generative AI projects by reviewing and labeling online content for accuracy and quality when project work is available.

Norway Belgium Sweden Denmark Finland Ireland Part-time Junior AI (Artificial Intelligence) Data Annotator
Up to $46k
Generative AI
12 hours, 23 minutes ago
Toloka

Creador(a) de Contenido POV para IA - Remoto

Toloka 251-1K Internet Software & Services

Proyecto remoto y flexible con una empresa de entrenamiento de IA que paga por grabaciones de actividades cotidianas realizadas desde casa para ayudar a entrenar futuros sistemas de IA y robótica.

United States Part-time Entry Level AI (Artificial Intelligence)
$21k-$21k
12 hours, 23 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers