RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

12 hours, 39 minutes ago
Sweden, Belgium, Denmark, Finland, Ireland, Norway
Contract
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 376

Description

  • Build realistic developer environments with codebases, infrastructure, tickets, documentation, and conversation history.
  • Design tasks from intermediate states of simulated environments and define what counts as a solved solution.
  • Write tests that verify agent solutions and accurately distinguish correct from incorrect approaches.
  • Review agent outputs, analyze failures, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation criteria to make tasks fair, robust, and challenging for AI coding agents.
  • Ensure tasks remain solvable by an AI agent while still testing real-world developer judgment.
  • Submit completed tasks by the deadline and meet the listed acceptance criteria.

Requirements

  • 5+ years of software development experience.
  • Experience with the core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Ability to work on a project-based basis rather than permanent employment.
  • Ability to complete tasks that are estimated at around 20 hours each, with flexibility to choose when and how to work.

Benefits

  • Up to $50/hr equivalent compensation, depending on level and pace.
  • Project-based flexible schedule with no fixed working hours.
  • Estimated 20 hours per task, allowing candidates to choose when and how to work.
  • Opportunity to work on AI evaluation projects for leading tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Leading Educators

Director of AI-Enabled Instruction, ELA

Leading Educators 51-250 Diversified Consumer Services

Leading Educators is hiring a Director of AI-Enabled Instruction, ELA to lead a multi-school literacy initiative and advance its broader AI-enabled ELA strategy through curriculum-aligned instructional design, coaching, and systems support.

United States Full-time Lead AI (Artificial Intelligence) Instructional Designer
$104k-$140k
Prototyping
11 hours, 54 minutes ago
Remote Recruitment

AI Integration Specialist - Italian Speaking

Remote Recruitment 11-50 Professional Services

AI Integration Specialist role at an Italian dental business coaching and consulting organisation, focused on implementing AI and automation solutions that improve efficiency, data use, and business performance across the company.

South Africa Italy Full-time Mid Level AI (Artificial Intelligence) AI Engineer
LLM Machine Learning Python SQL Tableau
12 hours, 24 minutes ago
Toloka

AI Trainer - Freelance Data Annotator

Toloka 251-1K Internet Software & Services

Toloka is hiring freelance annotators to support Generative AI projects by reviewing and labeling online content for accuracy and quality when project work is available.

Norway Belgium Sweden Denmark Finland Ireland Part-time Junior AI (Artificial Intelligence) Data Annotator
Up to $46k
Generative AI
12 hours, 24 minutes ago
Toloka

Creador(a) de Contenido POV para IA - Remoto

Toloka 251-1K Internet Software & Services

Proyecto remoto y flexible con una empresa de entrenamiento de IA que paga por grabaciones de actividades cotidianas realizadas desde casa para ayudar a entrenar futuros sistemas de IA y robótica.

United States Part-time Entry Level AI (Artificial Intelligence)
$21k-$21k
12 hours, 24 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers