Freelance Agent Evaluation Engineer

1 day, 6 hours ago
Part-time
Senior
Software Development
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services

Description

  • Build realistic developer environments that include a virtual company, codebase, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design tasks from intermediate states of the environment, including writing prompts and defining clear success criteria.
  • Ensure each task is solvable by an AI agent and reflects realistic developer work.
  • Write functional and integration tests that verify correct agent solutions and reject incorrect ones.
  • Review agent outputs, analyze failure cases, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation criteria until tasks are fair, robust, and neither too strict nor too lenient.
  • Evaluate how well AI coding agents handle real-world developer tasks.
  • Work within a project-based workflow where tasks are completed independently and submitted by deadline.

Requirements

  • 5+ years of experience in software development.
  • Experience with a core stack including Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Ability to work on project-based assignments rather than permanent employment.
  • Ability to estimate and complete tasks that may take around 20 hours each, depending on complexity.

Benefits

  • Up to $40/hr equivalent compensation, depending on level and pace.
  • Project-based work with flexible scheduling; you choose when and how to work.
  • Estimated 20 hours of work per task, with no fixed schedule requirement.
  • Payment upon successful task completion and acceptance.
  • Opportunity to work on leading tech company AI evaluation projects.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Mental Health Professionals - AI Training - Coventry, UK

Prolific 51-250 Professional Services

Prolific is hiring Mental Health Professionals to help train and evaluate AI models by reviewing psychology-related outputs and providing expert feedback as paid domain participants.

5 hours, 55 minutes ago

Data Entry Clerk - Graduates - AI Training - Colchester, UK

Prolific 51-250 Professional Services

Prolific is hiring Data Entry Specialists to help train and evaluate AI models by reviewing and annotating real-world data tasks for accuracy and quality.

Confluence CRM ERP Notion SQL
6 hours, 10 minutes ago

Process Automation Lead - Supply

Tripadvisor 1K-5K Consumer Services

Viator, a Tripadvisor company, is hiring a UK-based remote Process Automation Lead to drive AI- and automation-enabled transformation across its Supply organization and help scale global operations.

Agile Generative AI LLM
6 hours, 10 minutes ago

Integration & Test Engineer (Fairings)

SpaceX 10K-50K Aerospace & Defense

SpaceX is hiring an Integration & Test Engineer for its fairings program to support refurbishment, testing, and re-flight of reusable rocket hardware for customer and Starlink missions.

6 hours, 10 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers