RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

8 hours, 21 minutes ago
Israel, Jordan, Kuwait, Qatar, Saudi Arabia, Turkey
Part-time
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 117

Description

  • Build realistic developer environments that include a virtual company, codebase, infrastructure, and supporting context such as tickets, docs, and conversations.
  • Design coding tasks from intermediate states of those environments and define what counts as a solved solution.
  • Write tests that validate agent outputs, accepting all correct approaches and rejecting incorrect ones.
  • Review agent solutions, analyze failures, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation criteria to ensure tasks are fair, robust, and solvable by an AI agent.
  • Assess how well AI coding agents handle real-world developer tasks in simulated settings.

Requirements

  • 5+ years of experience in software development.
  • Experience with the core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
  • Experience writing tests, including functional and integration tests.
  • English proficiency at B2 level or higher.
  • CV must be submitted in English.
  • Availability to complete project tasks within the stated deadline.
  • Ability to work on a project-based, non-permanent basis.
  • Understanding of how to create challenging coding tasks for advanced AI models (preferred).

Benefits

  • Up to $50/hr equivalent compensation, depending on level and pace.
  • Project-based work with flexible scheduling.
  • Estimated 20 hours of work per task, depending on complexity.
  • Ability to choose when and how to work.
  • Payment upon completion of accepted tasks.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Prolific

AI Trainer - Mechanical Engineers - CAD Expertise - (Remote Advisory - France)

Prolific 51-250 Professional Services

Prolific is seeking Mechanical Engineers to join its remote expert network for CAD-focused AI research projects that capture real-world design and validation workflows.

France Contract Senior AI (Artificial Intelligence)
$25k-$250k
Linux Python
6 hours, 51 minutes ago
Webflow

Senior Software Engineer, Convert (Backend/Infra)

Webflow 251-1K Internet Software & Services

Webflow is hiring a remote-first Senior Software Engineer in Argentina to build and operate backend systems for its agentic recommendations and AI workflow platform, with a focus on reliability and infrastructure ownership.

Argentina Full-time Senior Software Engineer
Apache Airflow AWS CI/CD Kubernetes LLM MongoDB Node.js Python TypeScript
6 hours, 51 minutes ago
Validation Cloud

Senior Software Engineer, Backend

Validation Cloud 11-50 IT Services

Validation Cloud is hiring a Senior Backend Engineer to build scalable Node API and Web3 data services for its AI platform powering Web3 finance.

Anywhere Full-time Senior Software Engineer
AWS Azure Blockchain Docker GCP Go Kubernetes
7 hours, 6 minutes ago
Prolific

AI Trainer - Mechanical Engineers - CAD Expertise - (Remote Advisory - Germany)

Prolific 51-250 Professional Services

Prolific is seeking mechanical engineers to join a remote advisory talent pool supporting AI research by documenting real-world CAD design and validation workflows.

Germany Contract Senior AI (Artificial Intelligence)
$25k-$250k
Linux Python
7 hours, 6 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers