Freelance Agent Evaluation Engineer

2 hours, 49 minutes ago
Part-time
Senior
Software Development
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services

Description

  • Build realistic simulated developer environments with codebases, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design tasks from intermediate states of those environments, including crafting prompts and defining what counts as a solved task.
  • Write tests that validate AI agent solutions, accepting all correct approaches and rejecting incorrect ones.
  • Review agent solutions, analyze failures, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation criteria until tasks are fair, robust, and solvable by AI agents.
  • Help build a dataset used to assess how well AI coding agents handle real-world developer work.

Requirements

  • 5+ years of experience in software development.
  • Experience with the core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
  • Experience writing tests, including functional and integration tests.
  • English proficiency at B2 level or higher.
  • Submit a CV in English and indicate your English proficiency.
  • Ability to work on a project-based basis rather than as a permanent employee.
  • Ability to complete tasks by the deadline and meet the listed acceptance criteria.
  • Experience understanding model failure modes and designing challenging coding scenarios (implied by the role).

Benefits

  • Up to $30/hr equivalent compensation, depending on level and pace.
  • Flexible scheduling with self-directed work hours.
  • Project-based work with an estimated 20 hours per task.
  • Opportunity to work on AI evaluation projects for leading tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

AI Trainer - Kannada - Kolkata, India

Prolific 51-250 Professional Services

Prolific is hiring a remote, task-based AI Trainer for Kannada to evaluate text and voice data so AI models better understand natural Kannada speech and nuance.

2 hours, 20 minutes ago

AI Trainer - Kannada - San Diego, US

Prolific 51-250 Professional Services

Prolific is hiring a remote, task-based AI Trainer for Kannada to evaluate audio and text snippets so its AI models better understand the nuance and authenticity of the Kannada language.

2 hours, 20 minutes ago

AI Trainer - Kannada - Glasgow, UK

Prolific 51-250 Professional Services

Prolific is hiring remote task-based AI Trainers for Kannada to evaluate text and audio snippets and help improve how AI understands the nuance and naturalness of the Kannada language.

2 hours, 34 minutes ago

AI Trainer - Kannada - Chennai, India

Prolific 51-250 Professional Services

Prolific is hiring a remote, task-based AI Trainer in Kannada to evaluate AI-generated text and audio so its models better understand the language’s natural nuance and authenticity.

2 hours, 34 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers