RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

12 hours, 23 minutes ago
Australia
Part-time
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 376

Description

  • Build realistic developer environments that include a virtual company, codebase, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design coding tasks from intermediate states of the environment, including crafting the prompt and defining what counts as a solved task.
  • Ensure each task is solvable by an AI agent while remaining challenging and realistic.
  • Write functional and integration tests that verify agent solutions and distinguish valid approaches from incorrect ones.
  • Review agent solutions, analyze failures, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation criteria to make the dataset fair, robust, and effective for assessing AI coding agents.
  • Accept multiple valid solutions while avoiding tests that are overly strict or too lenient.

Requirements

  • 5+ years of experience in software development.
  • Experience with the core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Ability to work on a project-based, non-permanent engagement.
  • Willingness to complete qualification(s) before joining a project.
  • Ability to estimate and deliver tasks within a roughly 20-hour effort per task, by the required deadline.

Benefits

  • Up to $50/hr equivalent compensation, depending on level and pace.
  • Project-based flexible schedule with no fixed working hours.
  • Estimated ~20 hours of work per task, allowing you to choose when and how to work.
  • Opportunity to work on frontier AI evaluation projects for leading tech companies.
  • Payment upon successful task completion and acceptance.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Leading Educators

Director of AI-Enabled Instruction, ELA

Leading Educators 51-250 Diversified Consumer Services

Leading Educators is hiring a Director of AI-Enabled Instruction, ELA to lead a multi-school literacy initiative and advance its broader AI-enabled ELA strategy through curriculum-aligned instructional design, coaching, and systems support.

United States Full-time Lead AI (Artificial Intelligence) Instructional Designer
$104k-$140k
Prototyping
11 hours, 53 minutes ago
Remote Recruitment

AI Integration Specialist - Italian Speaking

Remote Recruitment 11-50 Professional Services

AI Integration Specialist role at an Italian dental business coaching and consulting organisation, focused on implementing AI and automation solutions that improve efficiency, data use, and business performance across the company.

South Africa Italy Full-time Mid Level AI (Artificial Intelligence) AI Engineer
LLM Machine Learning Python SQL Tableau
12 hours, 23 minutes ago
Toloka

AI Trainer - Freelance Data Annotator

Toloka 251-1K Internet Software & Services

Toloka is hiring freelance annotators to support Generative AI projects by reviewing and labeling online content for accuracy and quality when project work is available.

Norway Belgium Sweden Denmark Finland Ireland Part-time Junior AI (Artificial Intelligence) Data Annotator
Up to $46k
Generative AI
12 hours, 23 minutes ago
Toloka

Creador(a) de Contenido POV para IA - Remoto

Toloka 251-1K Internet Software & Services

Proyecto remoto y flexible con una empresa de entrenamiento de IA que paga por grabaciones de actividades cotidianas realizadas desde casa para ayudar a entrenar futuros sistemas de IA y robótica.

United States Part-time Entry Level AI (Artificial Intelligence)
$21k-$21k
12 hours, 23 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers