RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

1 day, 4 hours ago
Philippines
Part-time
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 218

Description

  • Build realistic developer environments that include a codebase, infrastructure, and supporting context such as tickets, docs, and conversations.
  • Design tasks from intermediate states of simulated environments and define clear success criteria for when a task is solved.
  • Write tests that validate agent solutions, accepting all correct approaches and rejecting incorrect ones.
  • Iterate on tasks and tests using QA feedback, reviewing agent solutions and refining failures.
  • Analyze where AI models fail on real developer tasks and use that insight to create challenging evaluations.
  • Ensure tasks are solvable by an AI agent while still being difficult for frontier models.
  • Submit completed tasks by the deadline and meet the stated acceptance criteria.

Requirements

  • 5+ years of experience in software development.
  • Experience with Python and FastAPI.
  • Experience with JavaScript/TypeScript and React.
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Ability to work on a project-based, non-permanent assignment.
  • Willingness to submit a CV in English and indicate English proficiency level.

Benefits

  • Up to $30/hr equivalent compensation, depending on level and pace.
  • Project-based work with flexibility to choose when and how you work.
  • Estimated 20 hours per task, allowing self-paced completion.
  • Payment upon completing tasks that are accepted.
  • Opportunity to work on AI evaluation projects for leading tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Toloka

Freelance Annotator (English) - AI Trainer

Toloka 251-1K Internet Software & Services

Toloka is seeking freelance annotators for project-based AI training work, reviewing and labeling data to help improve generative AI outputs when tasks are available.

Portugal Part-time Junior AI (Artificial Intelligence) Data Annotator
Generative AI
4 hours, 18 minutes ago
TherapyNotes

Senior Software Developer (Agentic Development)

TherapyNotes 51-250 Health Care Providers & Services

TherapyNotes LLC is hiring a Senior Software Developer to build internal AI- and automation-driven tools that improve software delivery across development, QA, product, and design teams.

United States Full-time Senior Software Engineer
$110k-$135k
Agile Angular C# CSS HTML JavaScript PostgreSQL Sass TypeScript
4 hours, 33 minutes ago
Delta Exchange

Quantitative Developer

Delta Exchange 11-50 Capital Markets

A Quantitative Developer at this trading firm will build and optimize low-latency trading systems and execution infrastructure that directly support trading performance and PnL.

India Full-time Mid Level Software Engineer
C++ Java Python
4 hours, 33 minutes ago
Pavago

AI-Native Marketing Coordinator

Pavago IT Services

Pavago’s client is hiring a remote AI-Native Marketing Coordinator to drive content, outreach, podcast production, events, and AI-powered marketing operations across a fast-growing startup.

Mexico Colombia Brazil Contract Junior AI (Artificial Intelligence) Marketing Specialist
Cloudflare Copywriting DNS GitHub Supabase
4 hours, 33 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers